{"id":"8a3de876-00d6-48c3-abdb-a74b28c92455","arxiv_id":"2412.08393","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"A self-iterative training loop that combines process reward models with ORPO improves reasoning accuracy of small language models on GSM8K and MBPP.","lead":"This paper trains small language models (up to 2 billion parameters) to reason better by having them generate their own solutions, then scoring each reasoning step and optimizing preferences in an iterative loop. It reports gains on math and code benchmarks, but the step-by-step grading comes from larger external models, not from the small model alone.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim depends on unvalidated inference-simulation step labels and an unreported threshold δ; Table 3 does not validate the actual simulation-trained PRM, so PRM bias can be amplified by ORPO's chosen-path SFT objective.","rationale":"The paper's strongest evidence is breadth: three SLMs, two reasoning domains, out-of-domain evaluations, and an ablation showing ORPO's relative-ratio loss contributes materially. The manual annotation effort in Appendix C is a genuine attempt to ground reward-model evaluation. However, that annotation is not connected to the simulation-labeled PRM actually used in SIPF. This is the load-bearing gap because the proposed method is defined by process feedback from simulated step labels; if those labels are miscalibrated, ORPO's chosen-path supervised term can reinforce exactly the wrong behavior, and self-iteration can magnify it. The concern is testable rather than fatal: the authors already have the human annotations needed to run the check. I am not recommending rejection; the paper can be accepted if the validation is supplied. Secondary issues—missing code/data, no error bars, and hyperparameters selected on the test set—are real but would only affect confidence in the reported magnitudes; the label-accuracy question goes to whether the proposed mechanism works at all.","tokens_in":16902,"tokens_out":10085,"duration_ms":114147,"concrete_test":"Use the 791 human-annotated GSM8K rationales from Appendix C as a held-out validation set. Recompute Eq. 2 step labels using the same deepseek-math simulator with K=8 across a grid of δ values, and report step-level agreement, precision, and recall against the human labels. Then rebuild D_simulate for one Gemma-2B iteration, retaining only steps where simulation and human labels agree, retrain the PRM, and re-run SIPF on GSM8K. Also report the final-answer correctness rate of PRM-chosen paths in the original D_pref. If the simulated labels show low human agreement, if a substantial fraction of chosen paths have wrong final answers, or if the human-filtered SIPF no longer beats SIOF or preserves the claimed 12.43-point improvement over SFT, the central claim is not supported. If agreement is high and the filtered result is unchanged, the concern is resolved.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The load-bearing premise is that the inference-simulation labels of Eq. 2 are accurate enough that the PRM trained on D_simulate (Eq. 3) yields reliable preference pairs (Eq. 6). Every reported SIPF gain passes through this premise, and it is the only step in the training loop without human-annotated ground truth. The paper never reports the threshold δ, never compares the Eq. 2 labels ci,j with the human step labels collected in Appendix C for the actual training data, and never reports the fraction of PRM-chosen τ^w that have a wrong final answer. Table 3 is not a substitute: as reported, it compares a PRM trained on human step-annotated rationales against an ORM, not the simulation-trained PRM used in SIPF against a human-trained PRM. Because ORPO's SFT term (Eq. 7) directly raises the probability of the chosen path τ^w, any systematic PRM bias—for example, favoring verbose or template-like steps regardless of mathematical validity—is imprinted into the SLM and can be amplified over Iter1–Iter3. The reported Iter1→Iter3 gains in Tables 1 and 2 could therefore reflect amplification of verifier bias rather than improved reasoning. The external 7B simulator/verifier makes this gap more important, not less, because the 'self-generated' signal is partly produced by a stronger model whose error structure is unmeasured.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SIPF ('self-iterative process feedback'), a fine-tuning and alignment procedure for small language models. The method first samples multiple reasoning paths from an SFT-initialized SLM, labels intermediate steps as correct or incorrect via Monte-Carlo-style inference simulation with an external 7B model, trains a process reward model on these simulated labels, scores sampled paths with the PRM to build chosen/rejected preference pairs, and aligns the SLM with ORPO. This is repeated for several iterations. Experiments on GSM8K/MMLU_Math (math) and MBPP/HumanEval (code) with TinyLlama-1.1B, Phi-1.5, and Gemma-2B report consistent gains over SFT, STaR, RFT, and SRF, plus ablations showing that the iterative loop, process feedback, and ORPO's ratio loss all contribute.","tokens_in":17223,"tokens_out":6994,"duration_ms":67030,"significance":"If the reported results hold, the paper makes a practical contribution: it shows that small models can be improved on multi-step reasoning without human step annotations, using an external simulator/verifier and ORPO. The use of multiple model families, the inclusion of a manual-annotation evaluation protocol in Appendix C, and the GPT-4-based rationale evaluation are strengths. The central evidence, however, is not yet conclusive because the process labels that drive the whole loop are unvalidated, OOD benchmarks are small, and no uncertainty quantification is provided.","major_comments":[{"comment":"The load-bearing premise of SIPF is that the simulation-derived step labels c_{i,j} in Eq. (2) are accurate enough that a PRM trained on D_simulate produces reliable preference pairs in Eq. (6). The paper never reports the threshold δ, never compares c_{i,j} against the human step annotations collected in Appendix C, and never evaluates the actual simulation-trained PRM. Table 3 is not a substitute: it compares a PRM trained on human step labels against an ORM, not the simulation-trained PRM used by SIPF. Because ORPO's SFT term (Eq. 7) directly raises the probability of the chosen path τ^w, a systematically biased PRM (e.g., one that favors verbose or template-like steps) would be imprinted into the SLM and could be amplified over Iter1–Iter3. Please report δ and η, evaluate the simulation-trained PRM on the 791-sample human-annotated set, and report the fraction of chosen τ^w whose final answer is wrong.","section":"2.2, Eq. (2); 2.3; 2.4; Table 3"},{"comment":"All experimental results are single runs without error bars, confidence intervals, or significance tests. The out-of-domain test sets are very small (MMLU_Math has 354 samples; HumanEval has 164), so differences of a few percentage points on OOD tasks (e.g., TinyLlama MMLU_Math 7.91 vs. 9.60 for Iter2 vs. Iter3) correspond to a handful of questions and may be noise. Please add bootstrap confidence intervals or significance tests and temper the OOD generalization claim accordingly.","section":"3.4; Tables 1 and 2"},{"comment":"Algorithm 1 sets M_k = ORPO(M, D_pref), i.e., it starts from the original pretrained model M rather than from M_{k-1}. This contradicts the description in Sections 2.1 and 2.5, where each iteration aligns the previous model. If this is a typo, please fix it; if it is intentional, the procedure is data accumulation with retraining from scratch, which changes the interpretation of the Iter1–Iter3 improvements.","section":"Algorithm 1, line 7"},{"comment":"The authors state that the best performance is selected from a predefined grid of learning rates and ORPO β weights, apparently evaluated on the test sets, and the selected values are not reported. This creates a risk of overfitting to GSM8K/MBPP test sets and makes the comparisons hard to reproduce. Please report the selected configuration for each model and task and, if possible, use a held-out validation split for hyperparameter selection.","section":"Appendix A.2, Table 5"}],"minor_comments":[{"comment":"The reward expression r(τ) = Σ_{j=0}^m \\hat c_{i,m} / m appears to contain an index error; the summand should presumably be \\hat c_{i,j}. As printed, the formula averages the final step's score m+1 times.","section":"2.3, Eq. (5)"},{"comment":"The sentence 'process feedback methods (SIPF and RFT) continue to improve' appears to name the wrong method; RFT is outcome-based and Figure 4's process variant is pRFT.","section":"4.5"},{"comment":"The word 'Accuarcy' should be 'Accuracy'.","section":"Table 4 caption"},{"comment":"The phrase 'can be incorporate into a single process' should read 'can be incorporated'.","section":"2.4"},{"comment":"The fact that the MBPP test set is re-formatted into six input formats per question should be stated in the main text, since Pass@1 computed under this protocol may not be directly comparable to the standard MBPP leaderboard.","section":"Appendix A.1, MBPP"}],"recommendation":"major_revision","confidential_remarks":"The main risk is the unvalidated PRM; the editor should weigh whether the authors can credibly claim 'self-generated process feedback' when the simulator and verifier are external 7B models. The paper also appears to overlap significantly with Math-Shepherd (Wang et al., 2024), which uses the same MCTS-style automatic step labeling; the distinction should be sharpened. If the authors provide the requested PRM validation and uncertainty estimates, the paper could be a solid empirical contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The new thing here is the specific pipeline: self-iterative sampling from the SLM, labeling step correctness via inference simulation, training a PRM on those labels, and aligning with ORPO on the resulting preference pairs. That exact combination is not in the cited literature, and the paper shows it works across three small models on GSM8K and MBPP, with mostly positive out-of-domain results. The ablation in Table 4 and the iteration curves in Figure 4 give real support for the claim that both process feedback and ORPO's relative-ratio loss matter. The paper also states its limitations plainly, including the ≤2B model scope and the resource overhead. That is honest work.\n\nThe soft spots are real but mostly fixable. The biggest one is the verifier. The paper never reports the threshold δ from Eq. 2, never reports η, and never validates the actual simulation-trained PRM used in the loop against the human-annotated step data they collected in Appendix C. Table 3 is not a substitute: it compares a separately trained human-annotated PRM against an ORM, not the simulation-trained verifier that generates all preference pairs. That matters because every SIPF gain passes through that verifier, and if it is biased—say, favoring verbose or superficially plausible steps—ORPO's SFT term imprints the bias and the self-iteration can amplify it. The paper does not show the simulation labels are accurate enough to rule this out.\n\nSecond, there are no error bars or significance tests anywhere. Some reported numbers fluctuate across iterations (Gemma-2B on MMLU_Math goes 25.98 -> 29.10 -> 28.25; HumanEval gains are small and non-monotonic), so a reviewer cannot tell which differences are stable. Third, hyperparameters appear to be selected on the test set: the paper says it picks the best learning rate and OR weight from a grid in Table 5 and leaves optimal search for future work. That inflates the reported gains. Finally, no code or data is released, which makes it hard to reproduce the pipeline even if the thresholds were reported.\n\nThe central direction holds up: with the caveats above, the method does beat SFT, STaR, RFT, and SRF in most settings, and the ablation gives credible evidence that the components matter. The verifier-validation gap is a genuine weakness but not a fatal one; it is addressable by reporting δ, evaluating the actual simulation-trained PRM on the Appendix C data, and adding error bars.\n\nThis paper deserves a serious referee. It is a solid empirical contribution to the small-model reasoning literature, and the key concerns are articulable and fixable. I would engage with it, though I would push for code release and a direct validation of the simulation-trained verifier before trusting the numbers.","headline":"A genuinely new combination of self-iteration, simulation-based process rewards, and ORPO that shows consistent gains on small models, but the load-bearing verifier is never validated and several key thresholds go unreported.","tokens_in":17755,"tokens_out":1983,"would_cite":false,"duration_ms":22448,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Small language models can learn to reason from step-level feedback they generate for themselves, and the paper shows this beats supervised fine-tuning and self-taught baselines on math and code reasoning benchmarks.","keywords":["small language models","reasoning","process reward model","self-iterative learning","preference optimization","ORPO","Chain-of-Thought","GSM8K"],"falsifier":"Hand-label the steps of a random sample of GSM8K reasoning paths, compute the agreement between those human labels and the simulation-based labels from the paper's Eq. 2, and retrain the full pipeline on the human labels alone; disagreement on a substantial fraction of steps, or failure of the human-labeled pipeline to reproduce the reported gains, would refute the central premise.","tokens_in":16682,"feed_emoji":"🧮","tokens_out":9803,"duration_ms":89068,"temperature":0.7,"pith_summary":"The paper argues that small language models (at or below 2 billion parameters) can improve multi-step reasoning without human step labels by generating their own positive and negative reasoning paths, labeling each step through repeated sampling simulation, and fine-tuning on the resulting preference pairs with an odds-ratio preference objective. On GSM8K the method raises Gemma-2B accuracy by 12.43 points over supervised fine-tuning, and on MBPP it raises Pass@1 by 3.95 points, while also improving out-of-domain performance on MMLU_Math and HumanEval. The central move is to replace outcome-only rewards with a process reward model trained on automatically generated step labels, and to align the model with ORPO rather than DPO, which the paper finds avoids suppressing the chosen responses. If the claim holds, small models can reason better using only their own samples and a strong off-the-shelf verifier, without expensive annotation.","feed_headline":"Self-scored reasoning steps lift 2B models by 12 points on GSM8K","feed_subtitle":"No human step labels needed: the model labels its own reasoning, builds preferences, and improves over iterations.","key_machinery":"The load-bearing mechanism is sampling-based inference simulation: from each intermediate step, the simulator samples several continuations, and the step is labeled correct only if more than a preset threshold number of simulated continuations reach the gold answer. Those labels train a process reward model, a verifier that assigns a correctness score to every step, and a path's reward is the mean of its step scores. Preference pairs with a reward gap above a margin are then used to align the model with ORPO, an odds-ratio preference objective that combines language-model loss on the chosen path with a term that raises the chosen path's odds relative to the rejected path.","core_discovery":"On the paper's own terms, the discovery is that a small language model can learn to reason by scoring its own steps: sample diverse reasoning paths from the current model, simulate continuations from each intermediate step to decide whether that step leads to the correct answer, train a process reward model on those step labels, build preference pairs from paths with separated reward scores, and align the model with ORPO. The paper reports that this self-iterative procedure outperforms supervised fine-tuning, self-taught sampling methods, and preference-based self-refine methods across mathematics and code benchmarks, and that the gains grow over iterations in most settings. It further claims that process feedback produces more reliable rationales than outcome feedback and that the DPO objective used in many self-refine baselines is a key source of their degradation because it lowers the probability of chosen responses.","pith_inferences":["The same loop should transfer to other verifiable multi-step domains, such as theorem proving or planning, by replacing the math verifier with a domain-specific checker; the paper only tests mathematics and code.","Because the verifier is a separate, stronger model, the loop's ceiling likely tracks verifier quality; varying verifier strength while holding everything else fixed would make that dependence measurable.","The head-to-head comparison suggests the optimizer choice (ORPO over DPO) may be as important as the step-level reward; an outcome-reward variant trained with ORPO would isolate whether the gains come from process labels or from the alignment objective.","The paper's limitation to models at or below 2B leaves open whether the loop scales; a natural test is running the same recipe on a 7B model and checking whether the per-iteration gains persist."],"forward_implications":["One iteration of the proposed method already beats supervised fine-tuning, self-taught methods, and outcome-based self-refine methods on both GSM8K and MBPP across the tested small models.","Additional iterations push GSM8K and MMLU_Math accuracy higher in most settings, whereas outcome-feedback iterative methods plateau or decline, suggesting process feedback keeps a longer improvement runway.","Process feedback produces rationales that an automated judge rates as more computationally correct, complete, reasonable, concise, and consistent than rationales from outcome feedback, even when both reach the same answer.","Removing process feedback, the odds-ratio loss, or both lowers accuracy on GSM8K, indicating that both the step signal and the ORPO objective contribute to the gains.","Process reward models trained on step labels evaluate reasoning steps more accurately than outcome reward models on a manually annotated GSM8K subset, supporting the choice of process supervision."],"supporting_citations":[{"why":"Provides the process reward model concept and the step-level supervision idea that the paper adapts to self-generated data.","marker":"Lightman et al., 2024"},{"why":"Supplies the ORPO objective that combines supervised fine-tuning and preference alignment in a single loss.","marker":"Hong et al., 2024"},{"why":"Defines the STaR iterative self-training baseline that the paper extends and compares against.","marker":"Zelikman et al., 2022"},{"why":"Defines RFT and the high-temperature rejection sampling strategy used to diversify self-generated reasoning paths.","marker":"Yuan et al., 2023"},{"why":"Provides the DPO objective used by the self-refine baselines whose degradation the paper analyzes.","marker":"Rafailov et al., 2023"},{"why":"Defines RPO, the outcome-feedback iterative preference optimization baseline that motivates the odds-ratio loss ablation.","marker":"Pang et al., 2024"},{"why":"Supplies the mathematics-specialized model used as simulator and process reward model in the math experiments.","marker":"Shao et al., 2024"},{"why":"Supplies the code-specialized model used as simulator and process reward model in the code experiments.","marker":"Guo et al., 2024"},{"why":"Provides the GSM8K dataset and the earlier idea of training verifiers for math word problems.","marker":"Cobbe et al., 2021"},{"why":"Provides the MBPP code generation benchmark used for in-domain code evaluation.","marker":"Austin et al., 2021"}],"fun_headline_variants":["Small LM self-scores its steps to reason better","2B model lifts GSM8K by 12 via self-iterative feedback","Self-process rewards improve small models on math and code","No external labels: small LM learns reasoning from itself","ORPO with self-supervised process feedback beats SFT"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that a reasoning step is genuinely correct whenever enough simulated continuations from that step reach the gold answer; if those automatic step labels are systematically noisy in a way that correlates with model mistakes, the reward model and the preference data trained on them will inherit that noise.","fun_headline_variants_meta":{"raw":{"variants":["Small LM self-scores its steps to reason better","2B model lifts GSM8K by 12 via self-iterative feedback","Self-process rewards improve small models on math and code","No external labels: small LM learns reasoning from itself","ORPO with self-supervised process feedback beats SFT"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000277,"raw_usage":{"total_tokens":1627,"prompt_tokens":901,"completion_tokens":726,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":517,"completion_tokens_details":{"reasoning_tokens":643}},"tokens_in":517,"tokens_out":726,"duration_ms":9087,"temperature":1.0,"reasoning_tokens":643,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:51:44.842394+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hand-label the steps of a random sample of GSM8K reasoning paths, compute the agreement between those human labels and the simulation-based labels from the paper's Eq. 2, and retrain the full pipeline on the human labels alone; disagreement on a substantial fraction of steps, or failure of the human-labeled pipeline to reproduce the reported gains, would refute the central premise.","supporting_citations":[],"review_version":1}