Pith. sign in

REVIEW 5 major objections 4 minor 2 cited by

ReST-RL combines reward-variance-filtered self-training with an annotation-free value model and MCTS decoding to improve LLM code reasoning.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

ReST-RL pairs reward-variance-filtered self-training (ReST-GRPO) with value-model-guided MCTS decoding (VM-MCTS) to improve LLM code reasoning.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Plausible training/decoding recipe, but the headline results may be inflated by training/eval overlap on BCB and APPS-500. the 5 major comments →

arxiv 2508.19576 v2 pith:UYBK7IZB submitted 2025-08-27 cs.AI cs.LG

ReST-RL: Achieving Accurate Code Reasoning of LLMs with Optimized Self-Training and Decoding

Sining Zhoubian , Dan Zhang , Jie Tang This is my paper
classification cs.AI cs.LG
keywords LLM reinforcement learningcode generationGRPOself-trainingvalue modelMonte Carlo tree searchprocess reward modeltest-time decoding
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that LLM code reasoning can be substantially improved by fixing two known weaknesses: GRPO's reward signals are too flat to learn from, and process reward models need expensive annotated data. Its answer is a two-stage paradigm: ReST-GRPO filters and assembles training prompts from the policy's own sampled solutions so that GRPO sees higher reward variance, and VM-MCTS trains a value model on MCTS-collected targets without human labels, then uses the value model to guide and verify decoding. Across four coding-focused LLMs and two general LLMs, the combined method beats naive GRPO, ReST-DPO, ORM, PRM, and ORM-MCTS on APPS-500, BigCodeBench, and the HumanEval/MBPP family. If the claim holds, code LLMs can be improved from small datasets (6,945 prompts) with both training and decoding gains stacking.

Core claim

The central claim, stated on the paper's own terms: LLM code reasoning improves when policy reinforcement and test-time verification are unified into one paradigm. ReST-GRPO first uses the current policy to sample solutions, keeps only prompts whose reward standard deviation is at least σ0 and whose best reward is at least r0, and augments those prompts with partial traces drawn from the best solution according to a geometric-type distribution; training on this assembled set with the GRPO objective increases reward variance and training efficiency. VM-MCTS then collects value targets by MCTS rollouts scored with a rule-based reward—no annotation—trains a value model on those targets, and at

What carries the argument

The load-bearing objects are two. (1) A reward-variance filter and data-assembly rule for ReST-GRPO: prompts are kept only when sampled rewards have std ≥ σ0 and best reward ≥ r0, and the best solution's partial states are added to prompts via p.m.f. p(a_{1..j}) = (1−α)/(1−α^{|A|})·α^{j−1}, which biases training toward earlier states while still sampling near the high-reward trace. (2) A value model trained on MCTS-generated value targets, deployed in an adapted MCTS that uses the VM's estimates in place of full rollouts; an appendix proves the value-based rollout estimator has no larger variance than complete-trace rollouts.

Load-bearing premise

The results depend on the evaluation benchmarks containing prompts that are genuinely new to the model: the training set Qtrain is drawn from BigCodeBench, DS-1000, and APPS, while evaluation also uses BigCodeBench and APPS-500, and the paper does not state that official train/test splits or deduplication were enforced.

What would settle it

Take the 6,945 training prompts and the evaluation prompts of BigCodeBench and APPS-500, and check for exact or near-duplicate matches (e.g., string hash plus embedding similarity). If any evaluation prompt matches a training prompt, rerun the comparison on a disjoint subset; the central claim would be weakened if ReST-RL's margin over GRPO shrinks or vanishes on the cleaned subset.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Because ReST-GRPO needs only a rule-based reward (test cases) and the policy's own samples, it can be applied wherever executable tests or similar checks exist, without human preference labels.
  • Same-step comparisons show ReST-GRPO training is more sample-efficient than GRPO and DAPO, so gains are not bought simply by more compute.
  • VM-MCTS's value model improves accuracy under every verification budget tested, from 1 to 100 samples, which suggests cheap small budgets also benefit.
  • Combining policy training with value-guided decoding gives additive gains—ReST-RL beats ReST-GRPO alone and VM-MCTS alone on average.
  • The method consistently improved across four base code LLMs and two general LLMs, with APPS-500 showing the largest relative gains, indicating hardest problems benefit most.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the reward-variance filter is the real driver, the same threshold logic could be used to schedule data collection: generate more samples only for prompts whose reward distribution is flat, and stop early on already-solved ones.
  • The VM is trained on the same policy's trajectories; out-of-domain prompts may yield unreliable value estimates, so a natural test is measuring VM-MCTS gains on benchmarks whose distribution differs from Qtrain (e.g., LeetCode-style problems).
  • A practical extension: use the VM's verification score inside ReST-GRPO's reward function for problems without public test cases, turning the two stages into a loop.
  • The reported gains on APPS-500 and BigCodeBench depend on evaluation prompts being disjoint from the 6,945 training prompts; a near-duplicate check is the first thing to run before trusting generalizability.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper proposes ReST-RL, a two-stage pipeline for improving code reasoning of LLMs. Stage 1, ReST-GRPO, samples N solutions per prompt, filters prompts by reward standard deviation and maximum reward, augments the selected prompts with partial prefixes of the best solution, and runs GRPO on the assembled data. Stage 2, VM-MCTS, trains a value model on MCTS-derived value targets without manual annotation and uses this model during decoding to guide an adapted MCTS search and to verify candidate solutions. Experiments on HumanEval, HumanEval+, MBPP, MBPP+, APPS-500, and BigCodeBench across several code-specialized and general LLMs report consistent gains over ReST-DPO, GRPO, ORM, PRM, and ORM-MCTS. The appendix contains a variance comparison between value-based rollouts and complete-trace rollouts.

Significance. If the reported results are clean, ReST-RL is a practically useful combination of reward-variance-aware self-training and MCTS-based value-model decoding, and it strengthens the evidence that code LLMs can be improved without expensive process-annotation pipelines. The paper's strengths include the use of external execution-based benchmarks, multiple base policies, iterative training comparisons, and a formal variance argument in Appendix A.4. However, the empirical claims rest on single-run evaluations and on benchmark hygiene that is currently not documented; these issues must be resolved before the significance of the method can be assessed.

major comments (5)
  1. [Section 4.1 (Training Datasets / Evaluation Benchmarks)] Qtrain is constructed from BigCodeBench, DS-1000, and APPS ("after filtering out training data without test cases"), while evaluation uses BigCodeBench and APPS-500. The manuscript never states that official train/test splits were used or that evaluation prompts were deduplicated against Qtrain. Since BigCodeBench is primarily an evaluation benchmark and APPS-500 is explicitly sampled from the APPS test set, the current text allows the training distribution to contain the exact BCB and APPS-500 prompts scored in Tables 1, 2, 4, and 5, as well as Figure 3. If overlap exists, the reported gains measure memorization rather than generalization. This is load-bearing for the central claim. Please report the split and deduplication procedure, overlap counts per benchmark, and, where necessary, rerun the main comparisons on disjoint held-out subsets.
  2. [Tables 1-3 and Figure 3 (statistical reliability)] All reported numbers are single-run point estimates with no error bars, multiple seeds, or significance tests. Many of the differences are small; for example, in Table 1, Qwen2.5-Coder-7B-Instruct ReST-GRPO first-iteration HumanEval is 0.872, identical to the base model, and DS-Coder-6.7b ReST-GRPO first-iteration HumanEval is 0.756, identical to ReST-DPO. The abstract's repeated claim that ReST-RL "significantly outperforms" baselines is therefore not supported by the evidence as presented. Please provide variance estimates (e.g., multiple seeds or bootstrap confidence intervals) at least for the headline tables.
  3. [Equation (8) and Section 4.2 (reward definition)] The training reward R_GRPO includes an "essential substring s" and a penalty based on n_redundant_char(A), but neither is defined. The substring identity, the set of substrings used, and the exact definition of redundant characters all affect the filtering step and the GRPO advantage. Without these definitions the experiments are not reproducible, and the term may inject task-specific priors beyond the test-case reward. Please specify the exact implementation (or point to the exact code path in the repository) and report sensitivity to omega_1 and omega_2.
  4. [Appendix A.4, Eq. (9) (variance proof assumptions)] The proof that value-based MC rollouts are more precise than complete-trace rollouts assumes that V_phi and R_lambda are unbiased estimators with equal variance (D[epsilon_V]=D[epsilon_R] and E[epsilon_V]=E[epsilon_R]=0). This assumption is asserted, not verified, for the trained VM and for the off-the-shelf ORM/PRM used in the experiments. The formal claim is therefore conditional on an untested property of the learned models. Please provide calibration/empirical variance comparisons of the actual value and reward models used, or weaken the claim accordingly.
  5. [Section 3.3.2 / Algorithm 2 (VM training data source)] The VM is trained on MCTS rollouts produced by the same policy that VM-MCTS later guides, using the same test-case reward as the only correctness signal. As a result, part of the VM-MCTS advantage over ORM/PRM baselines may reflect self-consistency of the policy rather than independent verification quality. The external test-case benchmarks prevent the main claim from being definitionally circular, but the contribution of value-based guidance is entangled with the reward used to collect value targets. Please compare against a variant that uses oracle/test-case rewards during MCTS rollouts, or otherwise quantify how much of the gain comes from the learned VM versus the reward signal itself.
minor comments (4)
  1. [Title/abstract formatting] The title appears as "REST-RL" in the header/abstract while the body uses "ReST-RL" and "ReST-GRPO". Please standardize the capitalization.
  2. [Figure 3(a) caption] The caption says the policy is "trained for 10k steps using two methods," but the figure compares three algorithms (ReST-GRPO, DAPO, GRPO). Please correct.
  3. [Table 4 / evaluation protocol] Table 4 mixes "Base" and "Base (greedy)" rows, and the main text does not explain how greedy decoding affects the APPS-500 Test Case Average metric. Please add a sentence describing the decoding protocol for each row.
  4. [Notation (Eq. 3 vs Eq. 4)] The symbol beta is used for the partial-state sampling ratio in Eq. (3) and again as the KL coefficient in Eq. (4). Please rename one of them to avoid ambiguity.

Circularity Check

2 steps flagged

Moderate circularity risk: Qtrain is drawn from the same BCB/APPS benchmarks later scored without a stated split or deduplication, and the reward-variance 'validation' is a restatement of the filtering rule.

specific steps
  1. fitted input called prediction [Section 4.1, Training Datasets and Evaluation Benchmarks]
    "After filtering out training data without test cases, we obtain a final train dataset Qtrain of 6945 coding prompts. ... we conduct test on six renowned benchmarks: HumanEval, HumanEval+, MBPP, MBPP+, BigCodeBench (BCB) and APPS. Among them, since the APPS test set is too large, we construct a new test set known as APPS-500 by randomly selecting 500 data from it."

    Qtrain is explicitly assembled from BigCodeBench and APPS, and the same BigCodeBench and APPS benchmarks are later used for evaluation. The paper never states that official train/test splits were enforced or that evaluation prompts were deduplicated against Qtrain. As written, the APPS-500 and BCB accuracy numbers may measure memorization of training prompts rather than generalization of code reasoning, so the central claim that ReST-RL significantly improves code reasoning on these benchmarks is not cleanly independent of the training input.

  2. self definitional [Section 3.2.2 / Figure 2]
    "For question prompts that the policy's solutions achieve a reward standard deviation less than a given threshold σ0, we filter them out from the train dataset ... we present in Figure 2 the comparison of training reward variances between ReST-GRPO and naive GRPO, which validates the effect of ReST-GRPO in improving the variance of rewards."

    ReST-GRPO is defined to discard prompts whose reward standard deviation falls below σ0, so the filtered training distribution necessarily has higher reward variance than GRPO's unfiltered distribution. Figure 2 is therefore a restatement of the filtering rule rather than an independent empirical confirmation. This is a supporting illustration, not the main benchmark result, so it contributes only minor circularity.

full rationale

The core algorithmic machinery—ReST-GRPO's sample-and-filter loop with a rule-based test-case reward, and VM-MCTS's value-target collection via MCTS—is self-contained and not definitionally circular. The main circularity risk comes from the evaluation setup: training data are drawn from BigCodeBench and APPS, and the same benchmark families are later scored, with no explicit statement of splits or deduplication. If evaluation prompts overlap Qtrain, the reported APPS-500 and BCB gains are fitted rather than predicted, though HumanEval/MBPP and cross-model trends remain external evidence. The reward-variance claim in Figure 2 is true by construction because low-variance prompts are filtered out by design. Self-citations to prior ReST-MCTS* work are not load-bearing, since the methods are fully described in this paper. Overall, this is partial rather than total circularity.

Axiom & Free-Parameter Ledger

9 free parameters · 5 axioms · 0 invented entities

The paper introduces no new physical or mathematical entities. The Value Model is a learned artifact, not a postulated entity. The main underlying assumptions are the reward signal's validity, the disjointness of training and evaluation data, and the equal-variance/unbiasedness condition for the theoretical variance-reduction claim. The ledger shows several hand-set hyperparameters and one unspecified reward feature (essential substring).

free parameters (9)
  • sigma_0 = 0.05
    Standard deviation threshold on group rewards for filtering prompts; hand-set in Section 4.2.
  • r_0 = 0.9
    Reward threshold for keeping a prompt's best solution; hand-set in Section 4.2.
  • alpha = 0.95
    Exponent factor in the partial-state sampling p.m.f. (Eq. 3); hand-set in Section 4.2.
  • beta = 0.5
    Ratio of partial states sampled from the best solution; hand-set in Section 4.2.
  • N = 30
    Number of solutions sampled per prompt per training iteration; chosen under resource constraints in Section 4.2.
  • omega_1, omega_2 = 1e-3, 1e-6
    Weights for the substring and redundant-character reward shaping terms in Eq. 8; hand-set in Section 4.2.
  • essential_substring_s = unspecified
    The 'essential substring' in Eq. 8 is not defined or extracted in the paper; it is a hidden per-prompt feature.
  • MCTS parameters = T=30, n=5, c=0.4 (train) / 0.1 (test), epsilon=0.1, v0=0
    Search-tree parameters for VM data collection and decoding, selected empirically in Appendix A.2.
  • learning_rate = 1e-7
    Learning rate for all policy training runs, stated in Section 4.2.
axioms (5)
  • domain assumption The test-case-based reward function R_base (Eq. 7) plus the shaping terms in Eq. 8 is a valid and sufficient training signal for code reasoning, and does not induce reward hacking.
    The whole ReST-GRPO training and VM value-target collection rely on this reward being a faithful proxy for code correctness. The paper explicitly avoids model-based rewards to reduce hacking risk, but does not validate the shaped reward against held-out correctness beyond the reported benchmarks.
  • domain assumption Evaluation benchmarks are disjoint from training data: no official train/test split or deduplication is stated for BigCodeBench and APPS in Section 4.1.
    Qtrain is built from BigCodeBench, DS-1000, and APPS; evaluation uses BigCodeBench and APPS-500. The absence of an explicit split statement makes this a load-bearing, unverified assumption.
  • ad hoc to paper The value model and reward model used in the Appendix A.4 proof are unbiased estimators with equal variance: D[epsilon_V] = D[epsilon_R] and E[epsilon_V] = E[epsilon_R] = 0.
    Equation 9 in Appendix A.4 postulates this directly as a 'reasonable approximation'. No empirical evidence is given that VM predictions are unbiased or that their error variance matches the reward model's. The variance-reduction conclusion follows only under this assumption.
  • domain assumption MCTS-collected value targets (Algorithm 2) approximate the true value function V_pi(S) under the policy, so training a VM on them is valid.
    Algorithm 2 uses MC rollouts and backpropagation to estimate state values. The accuracy of these estimates depends on the number of simulations, the exploration constant, and the reward function; the paper provides no validation of the convergence or bias of these targets.
  • domain assumption The transition to the next state is deterministic (appending a line to the solution), and the state/action formulation in Equations 1-2 is a valid Markov process representation for line-level decoding.
    Section 3.1 asserts determinism and defines actions as lines of text. This is a modeling choice that simplifies the MCTS and value model, but it does not account for stochastic sampling of continuations, which is the actual mechanism in rollout.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of ReST-RL: Achieving Accurate Code Reasoning of LLMs with Optimized Self-Training and Decoding." pith.science (2026). https://pith.science/paper/UYBK7IZB

@misc{pith2026250819576,
  author       = {Pith},
  title        = {Pith review of: ReST-RL: Achieving Accurate Code Reasoning of LLMs with Optimized Self-Training and Decoding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UYBK7IZB}},
  note         = {Machine review of arXiv:2508.19576}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

With respect to improving the reasoning accuracy of LLMs, the representative reinforcement learning (RL) method GRPO faces failure due to insignificant reward variance, while verification methods based on process reward models (PRMs) suffer from difficulties with training data acquisition and verification effectiveness. To tackle these problems, this paper introduces ReST-RL, a unified LLM RL paradigm that significantly improves LLM's code reasoning ability by combining an improved GRPO algorithm with a meticulously designed test time decoding method assisted by a value model (VM). As the first stage of policy reinforcement, ReST-GRPO adopts an optimized ReST algorithm to filter and assemble high-value training data, increasing the reward variance of GRPO sampling, thus improving the effectiveness and efficiency of training. After the basic reasoning ability of LLM policy has been improved, we further propose a test time decoding optimization method called VM-MCTS. Through Monte-Carlo Tree Search (MCTS), we collect accurate value targets with no annotation required, on which VM training is based. When decoding, the VM is deployed by an adapted MCTS algorithm to provide precise process signals as well as verification scores, assisting the LLM policy to achieve high reasoning accuracy. We conduct extensive experiments on coding problems to verify the validity of the proposed RL paradigm. Upon comparison, our approach significantly outperforms other reinforcement training baselines (e.g., naive GRPO and ReST-DPO), as well as decoding and verification baselines (e.g., PRM-BoN and ORM-MCTS) on well-known coding benchmarks of various levels (e.g., APPS, BigCodeBench, and HumanEval), indicating its power to strengthen the reasoning ability of LLM policies. Codes for our project can be found at https://github.com/THUDM/ReST-RL.

Figures

Figures reproduced from arXiv: 2508.19576 by Dan Zhang, Jie Tang, Sining Zhoubian.

Figure 1
Figure 1. Figure 1: Framework of ReST-RL. demonstrating notable performance on elementary programming tasks, LLMs continue to struggle with complex ones, such as those found in competitive programming platforms like Codeforces [4]. Improving reasoning abilities of LLMs. Previous methods that attempt to enhance the reasoning abilities of LLM mainly divide into two paths: LLM training and optimized decoding. For the first path,… view at source ↗
Figure 2
Figure 2. Figure 2: Distribution of standard deviation of group rewards during policy training. We can ob￾serve a clear distributional shift between two al￾gorithms. Training configurations are illustrated in Section 4.2, and the base LLM policy here is OpenCodeInterpreter-DS-6.7B. Considering that a partial state belonging to a high-reward solution may possibly be a good state for the policy to begin with, we extract a subse… view at source ↗
Figure 3
Figure 3. Figure 3: Test of training efficiency and budgeted verification. [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Performance of different base LLM policies when using ReST-RL and verification methods [PITH_FULL_IMAGE:figures/full_fig_p018_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Mental-R1: Aligning LLM Reasoning for Mental Health Assessment

    cs.AI 2026-06 unverdicted novelty 6.0

    CRPO extends group relative policy optimization with stage-dependent uncertainty modeling and reports a 10.4 percentage point weighted F1 gain over RL baselines across 8 mental health datasets.

  2. MARS$^2$: Scaling Multi-Agent Tree Search via Reinforcement Learning for Code Generation

    cs.AI 2026-04 unverdicted novelty 6.0

    MARS² integrates multi-agent collaboration with tree-structured search in RL to boost code generation by increasing exploratory diversity and using path-level group advantages for credit assignment.

Reference graph

Works this paper leans on

39 extracted references · 12 canonical work pages · cited by 2 Pith papers

  1. [1]

    Gpt-4 technical report, 2024

    OpenAI. Gpt-4 technical report, 2024

    2024

  2. [2]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y . Wu, Y . K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. Deepseek-coder: When the large language model meets programming – the rise of code intelligence, 2024

    2024

  3. [3]

    The lessons of developing process reward models in mathematical reasoning, 2025

    Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. The lessons of developing process reward models in mathematical reasoning, 2025

    work page 2025

  4. [4]

    Measuring coding challenge competence with apps, 2021

    Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with apps, 2021

    2021

  5. [5]

    Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions, 2025

    Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen Gong, Thong Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Yadav, Naman Jain, Alex Gu, Zhoujun Cheng, Jiawei Liu, Qian Liu, Zijian Wang, Biny...

    work page 2025

  6. [6]

    Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct, 2025

    Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, Yansong Tang, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct, 2025

    2025

  7. [7]

    Peiyi Wang, Lei Li, Zhihong Shao, R. X. Xu, Damai Dai, Yifei Li, Deli Chen, Y . Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce llms step-by-step without human annotations, 2024

    2024

  8. [8]

    Proximal policy optimization algorithms, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. 11

    work page 2017

  9. [9]

    Reinforced self-training (rest) for language modeling, 2023

    Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, Wolfgang Macherey, Arnaud Doucet, Orhan Firat, and Nando de Freitas. Reinforced self-training (rest) for language modeling, 2023

    2023

  10. [10]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024

    2024

  11. [11]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y . K. Li, Y . Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024

    2024

  12. [12]

    Beyond human data: Scaling self-training for problem-solving with language models.arXiv preprint arXiv:2312.06585, 2023

    Avi Singh, John D Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Peter J Liu, James Harrison, Jaehoon Lee, Kelvin Xu, Aaron Parisi, et al. Beyond human data: Scaling self-training for problem-solving with language models.arXiv preprint arXiv:2312.06585, 2023

    Pith/arXiv arXiv 2023

  13. [13]

    Rest-mcts*: Llm self-training via process reward guided tree search, 2024

    Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. Rest-mcts*: Llm self-training via process reward guided tree search, 2024

    2024

  14. [14]

    Solving math word problems with process- and outcome-based feedback, 2022

    Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process- and outcome-based feedback, 2022

    2022

  15. [15]

    Let’s verify step by step, 2023

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step, 2023

    2023

  16. [16]

    Don’t throw away your value model! generating more preferable text with value-guided monte-carlo tree search decoding, 2024

    Jiacheng Liu, Andrew Cohen, Ramakanth Pasunuru, Yejin Choi, Hannaneh Hajishirzi, and Asli Celikyilmaz. Don’t throw away your value model! generating more preferable text with value-guided monte-carlo tree search decoding, 2024

    work page 2024

  17. [17]

    Dapo: An open-source llm reinforcement learning system at scale, 2025

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Weinan Dai, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, W...

    2025

  18. [18]

    R1-vl: Learning to reason with multimodal large language models via step-wise group relative policy optimization, 2025

    Jingyi Zhang, Jiaxing Huang, Huanjin Yao, Shunyu Liu, Xikun Zhang, Shijian Lu, and Dacheng Tao. R1-vl: Learning to reason with multimodal large language models via step-wise group relative policy optimization, 2025

    work page 2025

  19. [19]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

    2021

  20. [20]

    Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals

    Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushm...

    work page 2022

  21. [21]

    Code with codeqwen1.5, April 2024

    Qwen Team. Code with codeqwen1.5, April 2024. 12

    work page 2024

  22. [22]

    Opencodeinterpreter: Integrating code generation with execution and refinement

    Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. Opencodeinterpreter: Integrating code generation with execution and refinement. arXiv preprint arXiv:2402.14658, 2024

    Pith/arXiv arXiv 2024

  23. [23]

    Tenenbaum, and Chuang Gan

    Shun Zhang, Zhenfang Chen, Yikang Shen, Mingyu Ding, Joshua B. Tenenbaum, and Chuang Gan. Planning with large language models for code generation, 2023

    work page 2023

  24. [24]

    Jiang, Jia Deng, Stella Biderman, and Sean Welleck

    Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q. Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics, 2024

    work page 2024

  25. [25]

    Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. Star: Bootstrapping reasoning with reasoning, 2022

    2022

  26. [26]

    Pspo*: An effective process-supervised policy optimization for reasoning alignment, 2024

    Jiawei Li, Xinyue Liang, Yizhe Yang, Chong Feng, and Yang Gao. Pspo*: An effective process-supervised policy optimization for reasoning alignment, 2024

    work page 2024

  27. [27]

    Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven C. H. Hoi. Coderl: Mastering code generation through pretrained models and deep reinforcement learning, 2022

    work page 2022

  28. [28]

    Chain-of-thought prompting elicits reasoning in large language models, 2023

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023

    2023

  29. [29]

    Griffiths, Yuan Cao, and Karthik Narasimhan

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models, 2023

    2023

  30. [30]

    Sra-mcts: Self-driven reasoning augmentation with monte carlo tree search for code generation, 2025

    Bin Xu, Yiguan Lin, Yinghao Li, and Yang Gao. Sra-mcts: Self-driven reasoning augmentation with monte carlo tree search for code generation, 2025

    work page 2025

  31. [31]

    Accessing gpt-4 level mathematical olympiad solutions via monte carlo tree self-refine with llama-3 8b, 2024

    Di Zhang, Xiaoshui Huang, Dongzhan Zhou, Yuqiang Li, and Wanli Ouyang. Accessing gpt-4 level mathematical olympiad solutions via monte carlo tree self-refine with llama-3 8b, 2024

    2024

  32. [32]

    Program synthesis with large language models, 2021

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021

    2021

  33. [33]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jia- jun Zhang, Bowen Yu, Kai Dang, et al. Qwen2. 5-coder technical report.arXiv preprint arXiv:2409.12186, 2024

    Pith/arXiv arXiv 2024

  34. [34]

    Qwen3 technical report, 2025

    Qwen Team. Qwen3 technical report, 2025

    2025

  35. [35]

    The llama 3 herd of models, 2024

    Meta. The llama 3 herd of models, 2024

    2024

  36. [36]

    Ds-1000: A natural and reliable benchmark for data science code generation, 2022

    Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Scott Wen tau Yih, Daniel Fried, Sida Wang, and Tao Yu. Ds-1000: A natural and reliable benchmark for data science code generation, 2022

    2022

  37. [37]

    Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation, 2023

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation, 2023

    2023

  38. [38]

    Reward hacking in reinforcement learning.lilianweng.github.io, Nov 2024

    Lilian Weng. Reward hacking in reinforcement learning.lilianweng.github.io, Nov 2024

    2024

  39. [39]

    Skywork-reward: Bag of tricks for reward modeling in llms.arXiv preprint arXiv:2410.18451, 2024

    Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. Skywork-reward: Bag of tricks for reward modeling in llms.arXiv preprint arXiv:2410.18451, 2024. 13 A APPENDIX A.1 MAINALGORITHMS We present the training data collection process of VM in Algorithm 2 and the assisted decoding algorithm of VM-M...

    Pith/arXiv arXiv 2024

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.