Pith. sign in

REVIEW 4 major objections 5 minor 12 references

Learning to Reason via Self-Iterative Process Feedback for Small Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Small language models can learn to reason from step-level feedback they generate for themselves, and the paper shows this beats supervised fine-tuning and self-taught baselines on math and code reasoning benchmarks.

desk verdict A genuinely new combination of self-iteration, simulation-based process rewards, and ORPO that shows consistent gains on small models, but the load-bearing verifier is never validated and several key thresholds go unreported. read the letter →

arxiv 2412.08393 v1 pith:E3DKONIU submitted 2024-12-11 cs.CL

Kaiyuan Chen , Jin Wang , Xuejie Zhang This is my paper · ORCID
classification cs.CL
keywords smalllanguagemodelsreasoningprocessrewardmodelself-iterativelearningpreferenceoptimizationORPOChain-of-ThoughtGSM8K
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that small language models (at or below 2 billion parameters) can improve multi-step reasoning without human step labels by generating their own positive and negative reasoning paths, labeling each step through repeated sampling simulation, and fine-tuning on the resulting preference pairs with an odds-ratio preference objective. On GSM8K the method raises Gemma-2B accuracy by 12.43 points over supervised fine-tuning, and on MBPP it raises Pass@1 by 3.95 points, while also improving out-of-domain performance on MMLU_Math and HumanEval. The central move is to replace outcome-only rewards with a process reward model trained on automatically generated step labels, and to align the model with ORPO rather than DPO, which the paper finds avoids suppressing the chosen responses. If the claim holds, small models can reason better using only their own samples and a strong off-the-shelf verifier, without expensive annotation.

What carries the argument

The load-bearing mechanism is sampling-based inference simulation: from each intermediate step, the simulator samples several continuations, and the step is labeled correct only if more than a preset threshold number of simulated continuations reach the gold answer. Those labels train a process reward model, a verifier that assigns a correctness score to every step, and a path's reward is the mean of its step scores. Preference pairs with a reward gap above a margin are then used to align the model with ORPO, an odds-ratio preference objective that combines language-model loss on the chosen path with a term that raises the chosen path's odds relative to the rejected path.

What would settle it

Hand-label the steps of a random sample of GSM8K reasoning paths, compute the agreement between those human labels and the simulation-based labels from the paper's Eq. 2, and retrain the full pipeline on the human labels alone; disagreement on a substantial fraction of steps, or failure of the human-labeled pipeline to reproduce the reported gains, would refute the central premise.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a small language model can learn to reason by scoring its own steps: sample diverse reasoning paths from the current model, simulate continuations from each intermediate step to decide whether that step leads to the correct answer, train a process reward model on those step labels, build preference pairs from paths with separated reward scores, and align the model with ORPO. The paper reports that this self-iterative procedure outperforms supervised fine-tuning, self-taught sampling methods, and preference-based self-refine methods across mathematics and code benchmarks, and that the gains grow over iterations in most settings. It further claims that process feedback produces more reliable rationales than outcome feedback and that the DPO objective used in many self-refine baselines is a key source of their degradation because it lowers the probability of chosen responses.

Load-bearing premise

The method assumes that a reasoning step is genuinely correct whenever enough simulated continuations from that step reach the gold answer; if those automatic step labels are systematically noisy in a way that correlates with model mistakes, the reward model and the preference data trained on them will inherit that noise.

Editorial extensions

If this is right

  • One iteration of the proposed method already beats supervised fine-tuning, self-taught methods, and outcome-based self-refine methods on both GSM8K and MBPP across the tested small models.
  • Additional iterations push GSM8K and MMLU_Math accuracy higher in most settings, whereas outcome-feedback iterative methods plateau or decline, suggesting process feedback keeps a longer improvement runway.
  • Process feedback produces rationales that an automated judge rates as more computationally correct, complete, reasonable, concise, and consistent than rationales from outcome feedback, even when both reach the same answer.
  • Removing process feedback, the odds-ratio loss, or both lowers accuracy on GSM8K, indicating that both the step signal and the ORPO objective contribute to the gains.
  • Process reward models trained on step labels evaluate reasoning steps more accurately than outcome reward models on a manually annotated GSM8K subset, supporting the choice of process supervision.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same loop should transfer to other verifiable multi-step domains, such as theorem proving or planning, by replacing the math verifier with a domain-specific checker; the paper only tests mathematics and code.
  • Because the verifier is a separate, stronger model, the loop's ceiling likely tracks verifier quality; varying verifier strength while holding everything else fixed would make that dependence measurable.
  • The head-to-head comparison suggests the optimizer choice (ORPO over DPO) may be as important as the step-level reward; an outcome-reward variant trained with ORPO would isolate whether the gains come from process labels or from the alignment objective.
  • The paper's limitation to models at or below 2B leaves open whether the loop scales; a natural test is running the same recipe on a 7B model and checking whether the per-iteration gains persist.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces SIPF ('self-iterative process feedback'), a fine-tuning and alignment procedure for small language models. The method first samples multiple reasoning paths from an SFT-initialized SLM, labels intermediate steps as correct or incorrect via Monte-Carlo-style inference simulation with an external 7B model, trains a process reward model on these simulated labels, scores sampled paths with the PRM to build chosen/rejected preference pairs, and aligns the SLM with ORPO. This is repeated for several iterations. Experiments on GSM8K/MMLU_Math (math) and MBPP/HumanEval (code) with TinyLlama-1.1B, Phi-1.5, and Gemma-2B report consistent gains over SFT, STaR, RFT, and SRF, plus ablations showing that the iterative loop, process feedback, and ORPO's ratio loss all contribute.

Significance. If the reported results hold, the paper makes a practical contribution: it shows that small models can be improved on multi-step reasoning without human step annotations, using an external simulator/verifier and ORPO. The use of multiple model families, the inclusion of a manual-annotation evaluation protocol in Appendix C, and the GPT-4-based rationale evaluation are strengths. The central evidence, however, is not yet conclusive because the process labels that drive the whole loop are unvalidated, OOD benchmarks are small, and no uncertainty quantification is provided.

major comments (4)
  1. [2.2, Eq. (2); 2.3; 2.4; Table 3] The load-bearing premise of SIPF is that the simulation-derived step labels c_{i,j} in Eq. (2) are accurate enough that a PRM trained on D_simulate produces reliable preference pairs in Eq. (6). The paper never reports the threshold δ, never compares c_{i,j} against the human step annotations collected in Appendix C, and never evaluates the actual simulation-trained PRM. Table 3 is not a substitute: it compares a PRM trained on human step labels against an ORM, not the simulation-trained PRM used by SIPF. Because ORPO's SFT term (Eq. 7) directly raises the probability of the chosen path τ^w, a systematically biased PRM (e.g., one that favors verbose or template-like steps) would be imprinted into the SLM and could be amplified over Iter1–Iter3. Please report δ and η, evaluate the simulation-trained PRM on the 791-sample human-annotated set, and report the fraction of chosen τ^w whose final answer is wrong.
  2. [3.4; Tables 1 and 2] All experimental results are single runs without error bars, confidence intervals, or significance tests. The out-of-domain test sets are very small (MMLU_Math has 354 samples; HumanEval has 164), so differences of a few percentage points on OOD tasks (e.g., TinyLlama MMLU_Math 7.91 vs. 9.60 for Iter2 vs. Iter3) correspond to a handful of questions and may be noise. Please add bootstrap confidence intervals or significance tests and temper the OOD generalization claim accordingly.
  3. [Algorithm 1, line 7] Algorithm 1 sets M_k = ORPO(M, D_pref), i.e., it starts from the original pretrained model M rather than from M_{k-1}. This contradicts the description in Sections 2.1 and 2.5, where each iteration aligns the previous model. If this is a typo, please fix it; if it is intentional, the procedure is data accumulation with retraining from scratch, which changes the interpretation of the Iter1–Iter3 improvements.
  4. [Appendix A.2, Table 5] The authors state that the best performance is selected from a predefined grid of learning rates and ORPO β weights, apparently evaluated on the test sets, and the selected values are not reported. This creates a risk of overfitting to GSM8K/MBPP test sets and makes the comparisons hard to reproduce. Please report the selected configuration for each model and task and, if possible, use a held-out validation split for hyperparameter selection.
minor comments (5)
  1. [2.3, Eq. (5)] The reward expression r(τ) = Σ_{j=0}^m \hat c_{i,m} / m appears to contain an index error; the summand should presumably be \hat c_{i,j}. As printed, the formula averages the final step's score m+1 times.
  2. [4.5] The sentence 'process feedback methods (SIPF and RFT) continue to improve' appears to name the wrong method; RFT is outcome-based and Figure 4's process variant is pRFT.
  3. [Table 4 caption] The word 'Accuarcy' should be 'Accuracy'.
  4. [2.4] The phrase 'can be incorporate into a single process' should read 'can be incorporated'.
  5. [Appendix A.1, MBPP] The fact that the MBPP test set is re-formatted into six input formats per question should be stated in the main text, since Pass@1 computed under this protocol may not be directly comparable to the standard MBPP leaderboard.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the SIPF training loop is grounded in gold-answer checks and external verifiers, and the reported gains are independent benchmark measurements.

full rationale

The paper's derivation chain is self-contained in the sense required by the circularity test. Reasoning paths are sampled from the SLM itself (Eq. 1), but their step-level labels are not taken from the model being evaluated: Eq. 2 defines c_{i,j} by comparing simulated final answers with the gold answer a*_i, and the simulator and verifier are fixed external models (deepseek-math-7b and deepseek-coder-6.7b) rather than quantities fitted to the final benchmark. The PRM trained by Eq. 4 therefore learns a mapping from externally grounded step labels, and Eq. 6 constructs preference pairs from PRM score margins; ORPO in Eq. 7 then optimizes the SLM against those pairs. The central claims (GSM8K accuracy, MBPP Pass@1, and out-of-domain MMLU_Math and HumanEval results) are measured on held-out benchmarks that are not used to define any parameter in the loop. Table 3 is a separate reward-model accuracy analysis on a manually annotated evaluation set, not a prediction derived from the fitted training loop, so it does not create a fitted-input-called-prediction problem. The paper contains self-citations (Chen et al. 2024a and Shen et al. 2023/2025), but they appear only in related-work background and do not justify any load-bearing premise. The unvalidated simulation threshold delta, the unreported verifier error propagation, and the possible amplification of verifier bias through ORPO are genuine robustness and correctness risks, but they are not circularity: the training signal still originates from gold answers and external models, not from the target result being defined in terms of the method's own fitted quantities.

Assumptions & free parameters 8 free parameters · 6 assumptions · 0 invented entities

The method introduces no new theoretical entities; it combines existing model components. The main unstated costs are the threshold values delta and eta, the choice of external verifier models, and the heuristic sample quotas. These are load-bearing for the self-iteration loop and are not independently validated.

free parameters (8)
  • learning_rate = 5e-5, 8e-5, 1e-4 (grid)
    Selected per task and model as best performance over a predefined grid with no reported validation split (Table 5, Appendix A.2).
  • ORPO relative ratio loss weight beta = 0.01, 0.1, 1 (grid)
    Weight for the odds ratio loss in Eq. (7); Figure 6 suggests 0.1 is suitable, but the value is chosen per run rather than fixed a priori.
  • step_correctness_threshold_delta = not reported
    Eq. (2) binarizes a step as correct only if the fraction of simulated rollouts reaching the gold answer exceeds delta; delta is never given.
  • preference_margin_eta = not reported
    Eq. (6) keeps a preference pair only if the reward difference exceeds eta; eta is never reported.
  • inference_simulation_sample_count_K = 8
    Eight simulated continuations per step are used for all tasks (Section 3.2); no ablation on K is reported.
  • sampling_temperature = 1.0 math, 0.7 code
    Chosen for reasoning-path sampling in each iteration (Section 3.2); no sensitivity analysis is provided.
  • per_iteration_sample_quota = decreasing by half or quarter per round
    Required number of positive and negative samples per question is set heuristically and reduced in later rounds (Appendix A.2).
  • reward_model_training_hyperparameters = batch size 40, learning rate 5e-5
    Separate reward model training settings are given in Appendix A.2 with no ablation.
assumptions (6)
  • domain assumption A step is correct if simulated rollouts starting from it reach the gold answer more often than threshold delta (Eq. 2).
    This is the only source of step-level supervision; it assumes Monte Carlo rollouts and gold answers are a faithful proxy for human step correctness, and delta is never reported.
  • domain assumption The PRM trained on simulated labels generalizes to unseen reasoning paths and produces reliable preference pairs.
    The entire preference dataset in Section 2.4 depends on verifier scores; no analysis of verifier error propagation into ORPO training is given.
  • domain assumption External 7B models (deepseek-math-7b-instruct/rl and deepseek-coder-6.7b-instruct) are capable simulators and reward models for SLMs.
    Section 3.2 assigns these models as generators of process feedback; if their capability is insufficient, the self-iteration signal is degraded.
  • domain assumption ORPO alignment on accumulated self-generated preference data improves reasoning monotonically without collapse.
    Algorithm 1 assumes repeated ORPO updates on growing preference sets are beneficial; the paper only reports aggregate accuracy, not training stability.
  • domain assumption QLoRA fine-tuning preserves enough model capacity for multi-step reasoning gains.
    All experiments use QLoRA (Section 3.2); no comparison to full fine-tuning is provided.
  • domain assumption The filtered MMLU_Math subset (354 samples) and HumanEval (164 samples) are valid out-of-domain test sets.
    Appendix A.1 describes aggressive filtering; the small sizes make point estimates noisy.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning to Reason via Self-Iterative Process Feedback for Small Language Models." pith.science (2026). https://pith.science/paper/E3DKONIU

@misc{pith2026241208393,
  author       = {Pith},
  title        = {Pith review of: Learning to Reason via Self-Iterative Process Feedback for Small Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/E3DKONIU}},
  note         = {Machine review of arXiv:2412.08393}
}
read the original abstract

Small language models (SLMs) are more efficient, cost-effective, and customizable than large language models (LLMs), though they often underperform in specific areas like reasoning. Past methods for enhancing SLMs' reasoning, such as supervised fine-tuning and distillation, often depend on costly external signals, resulting in SLMs being overly confident with limited supervision signals, thus limiting their abilities. Therefore, this study enables SLMs to learn to reason from self-iterative feedback. By combining odds ratio preference optimization (ORPO), we fine-tune and align SLMs using positive and negative signals generated by themselves. Additionally, we introduce process supervision for rewards in preference alignment by sampling-based inference simulation and process reward models. Compared to Supervised Fine-Tuning (SFT), our method improves the performance of Gemma-2B by 12.43 (Acc) on GSM8K and 3.95 (Pass@1) on MBPP. Furthermore, the proposed method also demonstrated superior out-of-domain generalization capabilities on MMLU_Math and HumanEval.

Figures

Figures reproduced from arXiv: 2412.08393 by the authors.

Figure 1
Figure 1. The conceptual diagram of the proposed self-iterative process feedback method against several previous [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overall framework of the proposed learning to reason from self-iterative process feedback. A single [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Comparison of the training processes of SFT-based self-taught, DPO-based SRF, and ORPO-based SIPF [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (6 more)
Figure 5
Figure 5. Figure 5: Automatic evaluation of the reliability of ra [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Comparison of performance with different [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: The performance of STaR at different iteration rounds on GSM8K (Acc) and MBPP (Pass@1). Iteration-0 [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: The case study of automatic evaluation of [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: In the GPT-4-based automatic evaluation of rationales, the prompt templates provide definitions for six [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: The guideline for annotating reasoning steps in sampled rationales from GSM8K. The guidelines provide [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 7 canonical work pages

  1. [1]

    Operational Correctness: The current step should involve correct basic calculations

    work page

  2. [2]

    Journal of Machine Learning Research, 25(70):1–53

    Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25(70):1–53. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Tim Dettmers, ...

    arXiv 2021

  3. [3]

    In The Twelfth In- ternational Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024

    Let’s verify step by step. In The Twelfth In- ternational Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. Open- Review.net. Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adámek, Eric Malmi, and Aliaksei Severyn. 2023. Teaching small language models to reason. In Pro- ceedings of the 61st Annual Meeting of the Asso...

    work page 2024

  4. [4]

    arXiv preprint arXiv:2404.19733

    Iterative reasoning preference optimization. arXiv preprint arXiv:2404.19733. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christo- pher D. Manning, Stefano Ermon, and Chelsea Finn

    arXiv

  5. [6]

    In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 9426–9439

    Math-shepherd: Verify and reinforce llms step- by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 9426–9439. Association for Computational Linguistics. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V . ...

    work page 2024

  6. [7]

    idx"：Problem ID

    Self-rewarding language models. arXiv preprint arXiv:2401.10020. Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. 2023. Scaling relationship on learning mathematical reasoning with large language models. arXiv preprint arXiv:2308.01825. Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Good- man. 2022....

    arXiv 2023

  7. [9]

    Any fabrication, distortion, or deviation from the facts is incorrect

    Factual Accuracy: The current step should not contradict the problem description. Any fabrication, distortion, or deviation from the facts is incorrect

    work page

  8. [10]

    Any contradictions in the reasoning are incorrect

    Logical Consistency: The current step should maintain logical coherence with previous reasoning steps or the problem itself. Any contradictions in the reasoning are incorrect

    work page

Show all 12 references
  1. [11]

    If a reasoning step meets the above criteria, it is considered correct; otherwise, it is incorrect

    Reasoning Completeness: If the current step is missing any key steps necessary for complete reasoning, the reasoning is considered incomplete. If a reasoning step meets the above criteria, it is considered correct; otherwise, it is incorrect. Please mark the correctness of the...

  2. [12]

    Self-Training Reasoning

    and potentially leading to hallucination is- sues (Rawte et al., 2023). Self-Training Reasoning. To overcome the lim- itations of manual annotation, STaR (Zelikman et al., 2022) was the first to propose leveraging the inherent language modeling capabilities of pre- trained lan...

    2024

  3. [2023]

    Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Sys- tems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023. Vipula Rawte,...

    2023 arXiv

  4. [2024]

    arXiv preprint arXiv:2402.14897

    Chain-of-thought unfaithfulness as disguised accuracy. arXiv preprint arXiv:2402.14897. Kaiyuan Chen, Jin Wang, and Xuejie Zhang. 2024a. Mathematical reasoning via multi-step self question- ing and answering for small language models. In CCF International Conference on Natural...

    2021 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.