Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

FreePRM: Training Process Reward Models Without Ground Truth Process Labels

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a process reward model trained only on final-answer correctness, with a buffer state to absorb pseudo-label noise, outperforms a fully step-supervised baseline on ProcessBench.

desk verdict A genuinely cheap way to train PRMs that reports a striking ProcessBench gain, but the paper's own theory says the method should collapse, and the benchmark numbers come from test-set-tuned hyperparameters. read the letter →

arxiv 2506.03570 v1 pith:MZ5EQGYX submitted 2025-06-04 cs.CL

Lin Sun , Chuang Liu , Xiaofeng Ma , Tao Yang , Weijia Lu , Ning Wu This is my paper
classification cs.CL
keywords processrewardmodelweaksupervisionpseudo-labelsbufferprobabilitymathematicalreasoningBenchoutcomelabelsbest-of-Nverification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FreePRM asks whether process reward models—models that score each reasoning step—can be trained without expensive step-level labels. The paper's answer is yes: it labels every step of a solution as correct when the final answer is correct and incorrect otherwise, then adds a third 'buffer' probability to each step so the model can express uncertainty and absorb the noise in those pseudo-labels. Trained this way on the Math-Shepherd dataset using only its problems and final answers, FreePRM reaches an average F1 of 53.0% on ProcessBench, beating the fully supervised Math-Shepherd PRM baseline (28.9%) by 24.1 points and several open-source PRMs trained on labeled data. The result matters because step-level annotation is the main bottleneck in building process supervision for multi-step reasoning; if the claim holds, outcome labels alone are a viable supervision signal.

What carries the argument

The central mechanism is the buffer probability $p^b_t$ attached to each reasoning step, alongside $p^r_t$ and $p^w_t$ with $p^r_t + p^w_t + p^b_t = 1$. For a pseudo label $\hat{y}_t$, the loss pushes $p^r_t + \beta_t p^b_t \to 1$ when $\hat{y}_t=1$ and $p^w_t + \beta_t p^b_t \to 1$ when $\hat{y}_t=0$, where $\beta_t \sim \mathrm{Bernoulli}(p^b_t)$ is a randomly sampled buffer factor. The buffer gives the model a neutral response for ambiguous steps, the random factor is meant to stop the model from simply setting $p^b_t=1$ everywhere, and the last step is trained with an amplified weight $\alpha_T > 1$ because it correlates most strongly with the final outcome. The paper also argues via a gradient calculation that the buffer acts as a quadratic regularizer that suppresses overfitting to noisy pseudo-labels.

What would settle it

Run FreePRM with the threshold and last-step weight fixed using only a training validation split, then score a held-out process-error set that was never inspected during development; if the F1 falls from 53.0% toward the 28.9% supervised baseline, the reported advantage is mostly test-set tuning. Separately, inspect the trained model's buffer probabilities: if many reasoning steps receive $p^b_t \approx 1$, the all-buffer collapse the paper says is impossible has in fact occurred.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes a training scheme for process reward models that uses only binary final-answer labels. The scheme generates pseudo step labels by assuming all steps are correct for a correct final answer and all steps incorrect otherwise, and it prevents this crude labeling from wrecking training by predicting three probabilities per step—right, wrong, and buffer—that sum to one. The buffer absorbs label noise, a random buffer factor prevents the model from collapsing into always predicting 'buffer', and extra weight on the last step anchors the prediction to the known outcome. The paper reports 53.0% average F1 on ProcessBench, a 24.1-point improvement over the fully supervised baseline Qwen2.5-Math-7B-Math-Shepherd-PRM (28.9%), and comparable or better best-of-N verification on MATH-500 than PRMs trained on labeled process data.

Load-bearing premise

The headline result assumes that the decision threshold and last-step weight, chosen by looking at ProcessBench scores, would remain the right choices on new data, and that the random-buffer training rule really prevents the model from collapsing into always predicting 'unsure'.

Editorial extensions

If this is right

  • Step-level annotation stops being a hard requirement: any dataset with final-answer correctness can be used to train a process reward model.
  • On ProcessBench, outcome-only training beats full step supervision from the same source data, suggesting that pseudo-label noise can be managed better than curated automatic step labels in this setting.
  • Best-of-N verification on MATH-500 matches or exceeds labeled-PRM baselines, especially at low sample counts, so the method is usable as a verifier without step labels.
  • The method inherits the scalability of weak supervision: 20% of the training data still yields 49.1% F1, so PRM training is feasible with limited data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the buffer probability could be read as a learned abstention channel: a step with high $p^b_t$ is one the model refuses to judge, and using $p^b_t$ directly for selective prediction or human-routing is a natural test the paper does not run.
  • The same three-way labeling could be transferred to non-mathematical verifiers, e.g., code or proof steps, where a final test or compiler check supplies weak outcome labels.
  • Because the right-score threshold (0.9) and last-step weight (3.0) were selected by inspecting ProcessBench, the 53.0% figure is likely an optimistic estimate of what the method would achieve on a new benchmark; the paper does not report a validation-split protocol.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes FreePRM, a weakly supervised framework for training process reward models using only binary final-outcome labels. It generates pseudo step-level labels from final correctness, adds a third 'buffer' probability to each step, and introduces a Bernoulli-sampled 'random buffer factor' plus an extra last-step loss weight to counter label noise and buffer collapse. The central empirical claim is that FreePRM reaches 53.0% average F1 on ProcessBench, outperforming a Math-Shepherd-trained PRM baseline (28.9%) by 24.1 points, with a code release and comparisons on BoN MATH-500 verification.

Significance. If the empirical result were cleanly established, the paper would make a useful contribution: it would show that competitive process reward models can be trained without step-level annotations, lowering annotation cost. The paper also provides a plausible and simple framework (pseudo-labeling plus a soft buffer class) that is easy to build on. However, the theoretical justification for the key anti-collapse mechanism is reversed, and the headline benchmark numbers depend on hyperparameters selected on the test set itself. The reported 53.0% F1 is therefore not yet a trustworthy out-of-sample result. The code release and the breadth of ProcessBench and BoN comparisons are strengths, but the absence of error bars and the undisclosed handling of the discrete Bernoulli sample leave the central claim in need of substantial revision.

major comments (4)
  1. [§2.2, Eq. (12) and Eq. (5)] Theorem 2.2 contains a sign error that reverses its conclusion. Equation (12) gives dE[L_t]/dp_b = -[log((p_r+p_b)/p_r) + p_b/(p_r+p_b)]. Near p_b=1, with p_r=epsilon, this is approximately -(log(1/epsilon)+1), which is negative and diverges to -infinity. Under gradient descent, a negative gradient increases p_b, pushing the model toward the all-buffer solution, not away from it. In fact, Eq. (5) has a global optimum at p_b=1 for every step: when beta_t=1 almost surely, p_r+beta_t p_b = 1 and p_w+beta_t p_b = 1, so both log terms are zero for both pseudo-labels. The proof's final sentence ('pushes p_b away from 1') is therefore contradicted by its own calculation. The paper must either correct this analysis, or disclose the actual training objective used in the experiments; as written, the mechanism claimed to prevent collapse is not established.
  2. [Appendix C and Table 4] The headline ProcessBench F1 is not a clean out-of-sample measurement. Appendix C states that the right-score threshold was selected to optimize F1 on ProcessBench and reports that the optimal threshold is 0.9, while Table 4 scans the last-step weight over 1.0, 3.0, 5.0, and 7.0 and appears to select 3.0 based on ProcessBench performance. No held-out validation set or nested hyperparameter-selection procedure is described. The authors should report results for hyperparameters chosen on a separate validation split, or provide the full threshold/weight curves so readers can judge the sensitivity of the 53.0% claim.
  3. [§3.2 and §3.3] The main result is based on a single training run per configuration, and no error bars, confidence intervals, or repeated-seed results are reported. This matters given the large swings in Table 4 (average F1 from 34.3 to 49.1 and back to 31.4 as the weight changes) and the non-monotonic data-fraction results in Table 2 (49.1 at 20%, 41.7 at 40%, 44.3 at 60%, 46.4 at 80%, 53.0 at 100%). The authors should add at least a small number of seeds or otherwise quantify the variability before the claimed advantage over fully supervised baselines can be assessed.
  4. [§3.3, Tables 3 and 4] The ablations in Tables 3 and 4 are run with 20% of the training data, while the headline 53.0% F1 is obtained with 100% of the data. The text does not state whether the alpha_T=3.0 and threshold values used in the full-data model were the same as those selected in the 20%-data ablations, or whether they were re-tuned. This makes it difficult to connect the ablations to the main result and to reproduce the full-data configuration. Please clarify the exact hyperparameters and selection procedure used for the final model.
minor comments (6)
  1. [§2.2, Theorem 2.2 proof] The proof contains typographical errors: 'frist term' and 'seconde term' should be 'first term' and 'second term'.
  2. [Figure 3 caption] In the caption, 'we pich the final step reward' should be 'we pick the final step reward'.
  3. [Appendix C] The appendix refers to 'PressBench' rather than 'ProcessBench', and the evaluation description should state explicitly that only the predicted right probability is used for error detection.
  4. [Table 3 and Figure 4] Table 3 and Figure 4 contain corrupted '/uni...' tokens that appear to be PDF encoding artifacts; these should be fixed to display the actual table and figure.
  5. [Abstract and §3.2] The text describes the Math-Shepherd-trained baseline as 'fully supervised', but Math-Shepherd process labels are automatically generated rather than human-annotated; 'fully supervised' should be qualified to avoid overstatement.
  6. [Abstract] The '+24.1% improvement' should be stated as '+24.1 percentage points' to distinguish an absolute F1 difference from a relative improvement.

Circularity Check

1 steps flagged · score 5.0 of 10

The headline ProcessBench F1 is partly fitted: the right-score threshold (0.9) and last-step weight (3.0) are selected by scanning ProcessBench itself, so the 53.0% result is a tuned maximum rather than a clean out-of-sample measurement; the no-collapse theorem is separately invalid, but that is a correctness flaw, not circularity.

  1. fitted input called prediction [Abstract 53.0% claim; Section 3.3 Table 4; Appendix C Figure 4]
    "Appropriate last-step weight boosts performance of FreePRM. ... increasing the weight from 1.0 to 3.0 leads to substantial improvements, with the average F1 score rising from 34.3 to 49.1. ... Figure 4: Influence of right score threshold selection on overall performance. ... The optimal F1 performance is achieved when the threshold is set to 0.9."

    Table 4 varies the last-step weight and reports ProcessBench F1; Figure 4 varies the right-score threshold and reports ProcessBench F1. The paper then adopts weight=3.0 and threshold=0.9 and reports the same benchmark's average F1 (53.0%) as its headline result in the Abstract and Table 1. Thus the headline number is the selected maximum of a two-parameter scan performed on the evaluation set itself. No separate validation split or fixed a-priori evaluation protocol is described. The reported performance is therefore not an independent measurement of the method; it is partly constructed by the same benchmark it claims to predict, making the 'prediction' statistically forced rather than out-of-sample.

full rationale

The FreePRM training scheme itself—pseudo step labels derived from final outcomes plus a three-way right/wrong/buffer prediction with a sampled buffer factor—is an externally proposed training objective and is not derived from ProcessBench. There is no load-bearing self-citation chain or imported uniqueness theorem; the buffer idea is explicitly credited to the neutral labels of PRM800K [10], so it is acknowledged inspiration rather than a disguised renaming. The circularity burden is therefore confined to the evaluation protocol: the right-score threshold and last-step weight are chosen by scanning ProcessBench, and the same ProcessBench average F1 is advertised as the central contribution. This is a fitted-input-called-prediction pattern and supports a moderate score. Separately, Theorem 2.2 contains a non-circular but serious derivation error: Eq. 12 yields a negative derivative of the expected loss with respect to p_b, so gradient descent would increase p_b toward the all-buffer optimum, the opposite of the proof's conclusion. I flag that as a correctness risk that independently undermines the claimed no-collapse guarantee, but it is not a form of circularity. Because the central approach still rests on an independent training framework and external baseline comparisons, the score is 5 rather than 6-8.

Assumptions & free parameters 2 free parameters · 4 assumptions · 1 invented entities

The method rests on two free hyperparameters tuned to the benchmark and on several unproven modeling assumptions, including the effectiveness of the pseudo-label rule and the stability of the random buffer factor. The buffer probability is an invented construct with no independent evidence outside the paper's own experiments.

free parameters (2)
  • right-score threshold = 0.9
    Selected on ProcessBench to maximize F1, as shown in Figure 4.
  • last-step weight α_T = 3.0
    Tuned on ProcessBench; Table 4 shows best average F1 at weight 3.0.
assumptions (4)
  • domain assumption Pseudo-label rule: all steps share the final outcome label (Eq. 1).
    Assumes a correct final answer implies all steps are correct and an incorrect final answer implies all are incorrect, which is known to be noisy.
  • domain assumption Buffer probability absorbs label noise without losing discriminative signal.
    Section 2.1 states this as motivation; no proof or independent evidence is given.
  • ad hoc to paper The Bernoulli-sampled buffer factor β_t has the gradient behavior described in Theorem 2.2.
    The proof appears to contain a sign error: a negative gradient would push p_b toward 1, not away, so the claimed instability is not established.
  • domain assumption Probabilities p_r, p_w, p_b are treated as independent variables subject only to the sum constraint in gradient derivations.
    The model outputs are softmax functions of logits, so gradients w.r.t. logits couple the three probabilities; the paper's derivations ignore this coupling.
invented entities (1)
  • Buffer probability p_b
    purpose: A third output class that represents uncertainty and absorbs noise from pseudo-labels.
    Introduced in Eqs. 2-5; only internal ablations support it, no external validation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FreePRM: Training Process Reward Models Without Ground Truth Process Labels." pith.science (2026). https://pith.science/paper/MZ5EQGYX

@misc{pith2026250603570,
  author       = {Pith},
  title        = {Pith review of: FreePRM: Training Process Reward Models Without Ground Truth Process Labels},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MZ5EQGYX}},
  note         = {Machine review of arXiv:2506.03570}
}
read the original abstract

Recent advancements in Large Language Models (LLMs) have demonstrated that Process Reward Models (PRMs) play a crucial role in enhancing model performance. However, training PRMs typically requires step-level labels, either manually annotated or automatically generated, which can be costly and difficult to obtain at scale. To address this challenge, we introduce FreePRM, a weakly supervised framework for training PRMs without access to ground-truth step-level labels. FreePRM first generates pseudo step-level labels based on the correctness of final outcome, and then employs Buffer Probability to eliminate impact of noise inherent in pseudo labeling. Experimental results show that FreePRM achieves an average F1 score of 53.0% on ProcessBench, outperforming fully supervised PRM trained on Math-Shepherd by +24.1%. Compared to other open-source PRMs, FreePRM outperforms upon RLHFlow-PRM-Mistral-8B (28.4%) by +24.6%, EurusPRM (31.3%) by +21.7%, and Skywork-PRM-7B (42.1%) by +10.9%. This work introduces a new paradigm in PRM training, significantly reducing reliance on costly step-level annotations while maintaining strong performance.

Figures

Figures reproduced from arXiv: 2506.03570 by the authors.

Figure 1
Figure 1. FreePRM introduces buffer area be￾tween wrong and right, along with predicting prob￾abilities of right and wrong, it also predicts buffer probability. It reflects the ambiguity of uncertain reasoning steps, and helps absorb noise from unre￾liable pseudo-labels. Unlike Outcome Reward Models (ORMs) [2], which evaluate only the final result, Process Re￾ward Models (PRMs) provide fine-grained feed￾back at each reasoning… view at source ↗
Figure 2
Figure 2. The input consists of question and its solution. The solution is divided into multiple steps, [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. BoN results with different generation models on MATH-500. we generate 128 solutions for [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Influence of right score threshold selection on overall performance. As the threshold [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. PaLMR: Towards Faithful Visual Reasoning via Multimodal Process Alignment

    cs.CV 2026-02 conditional novelty 5.0 of 10

    By gating the reinforcement-learning reward on a binary visual-fidelity score, PaLMR lifts Qwen2.5-VL-7B to 70.9 on HallusionBench and 59.3 on MMMU, surpassing its GRPO baseline.

Reference graph

Works this paper leans on

30 extracted references · 13 canonical work pages · cited by 1 Pith paper

  1. [1]

    Alphamath almost zero: Process super- vision without process

    Guoxin Chen, Minpeng Liao, Chengxi Li, and Kai Fan. Alphamath almost zero: Process super- vision without process. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, 8 Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang, editors,Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024...

    work page 2024

  2. [2]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    arXiv

  3. [3]

    Process reinforcement through implicit rewards.arXiv preprint arXiv:2502.01456, 2025

    Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, Jiarui Yuan, Huayu Chen, Kaiyan Zhang, Xingtai Lv, Shuo Wang, Yuan Yao, Xu Han, Hao Peng, Yu Cheng, Zhiyuan Liu, Maosong Sun, Bowen Zhou, and Ning Ding. Process reinforcement through implicit rewards.arXiv preprint arXiv:2502.01456, 2025

    arXiv 2025

  4. [4]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, and et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    arXiv 2024

  5. [5]

    LLM critics help catch bugs in mathematics: Towards a better mathematical verifier with natural language feedback

    Bofei Gao, Zefan Cai, Runxin Xu, Peiyi Wang, Ce Zheng, Runji Lin, Keming Lu, Junyang Lin, Chang Zhou, Wen Xiao, Junjie Hu, Tianyu Liu, and Baobao Chang. LLM critics help catch bugs in mathematics: Towards a better mathematical verifier with natural language feedback. arXiv preprint arXiv:2406.14024, 2024

    arXiv 2024

  6. [6]

    Measuring mathematical problem solving with the MATH dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Joaquin Vanschoren and Sai-Kit Yeung, editors,Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021...

    work page 2021

  7. [7]

    Qwen2.5-coder technical report.arXiv preprint arXiv:2409.12186, 2024

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Ji- ajun Zhang, Bowen Yu, Kai Dang, An Yang, Rui Men, Fei Huang, Xingzhang Ren, Xu- ancheng Ren, Jingren Zhou, and Junyang Lin. Qwen2.5-coder technical report.arXiv preprint arXiv:2409.12186, 2024

    arXiv 2024

  8. [8]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b.arXiv preprint arXiv:23...

    arXiv

Show all 30 references
  1. [9]

    Process reward model with q-value rankings.arXiv preprint arXiv:2410.11287, 2024

    Wendi Li and Yixuan Li. Process reward model with q-value rankings.arXiv preprint arXiv:2410.11287, 2024

    2024 arXiv

  2. [10]

    Let’s verify step by step

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11...

    2024

  3. [11]

    Autopsv: Automated process-supervised verifier

    Jianqiao Lu, Zhiyang Dou, Hongru Wang, Zeyu Cao, Jianbo Dai, Yunlong Feng, and Zhijiang Guo. Autopsv: Automated process-supervised verifier. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang, editors, Advances in...

    2024

  4. [12]

    Improve mathematical reasoning in language models by automated process supervision.arXiv preprint arXiv:2406.06592, 2024

    Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, and Abhinav Rastogi. Improve mathematical reasoning in language models by automated process supervision.arXiv preprint arXiv:2406.06592, 2024

    2024 arXiv

  5. [13]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y . K. Li, Y . Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024. 9

    2024 arXiv

  6. [14]

    Skywork-o1 open series

    Team Skywork. Skywork-o1 open series. https://huggingface.co/Skywork, November

  7. [15]

    Scaling LLM test-time compute op- timally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314, 2024

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute op- timally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314, 2024

    2024 arXiv

  8. [16]

    Q*: Improving multi-step reasoning for llms with deliberative planning.arXiv preprint arXiv:2406.14283, 2024

    Chaojie Wang, Yanchen Deng, Zhiyi Lv, Zeng Liang, Jujie He, Shuicheng Yan, and Bo An. Q*: Improving multi-step reasoning for llms with deliberative planning.arXiv preprint arXiv:2406.14283, 2024

    2024 arXiv

  9. [17]

    Math-shepherd: Verify and reinforce llms step-by-step without human annotations

    Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce llms step-by-step without human annotations. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors,Proceedings of the 62nd Annual Meetin...

    2024

  10. [18]

    Multi-step problem solving through a verifier: An empirical analysis on model-induced process supervision

    Zihan Wang, Yunxuan Li, Yuexin Wu, Liangchen Luo, Le Hou, Hongkun Yu, and Jingbo Shang. Multi-step problem solving through a verifier: An empirical analysis on model-induced process supervision. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors,Findings of the Asso...

    2024

  11. [19]

    Training large language models for reasoning through reverse curriculum reinforcement learning

    Zhiheng Xi, Wenxiang Chen, Boyang Hong, Senjie Jin, Rui Zheng, Wei He, Yiwen Ding, Shichun Liu, Xin Guo, Junzhe Wang, Honglin Guo, Wei Shen, Xiaoran Fan, Yuhao Zhou, Shihan Dou, Xiao Wang, Xinbo Zhang, Peng Sun, Tao Gui, Qi Zhang, and Xuanjing Huang. Training large language mo...

    2024

  12. [20]

    Evaluating mathematical reasoning beyond accuracy

    Shijie Xia, Xuefeng Li, Yixin Liu, Tongshuang Wu, and Pengfei Liu. Evaluating mathematical reasoning beyond accuracy. In Toby Walsh, Julie Shah, and Zico Kolter, editors,AAAI-25, Sponsored by the Association for the Advancement of Artificial Intelligence, February 25 - March 4...

    2025

  13. [21]

    An implementation of generative prm

    Wei Xiong, Hanning Zhang, Nan Jiang, and Tong Zhang. An implementation of generative prm. https://github.com/RLHFlow/RLHF-Reward-Modeling, 2024. Accessed: 2025-05-11

    2024

  14. [22]

    Qwen2.5 technical report.arXiv preprint arXiv:2412.15115, 2024

    An Yang, Baosong Yang, Beichen Zhang, and et al. Qwen2.5 technical report.arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  15. [23]

    Qwen2.5-math technical report: Toward mathematical expert model via self-improvement.arXiv preprint arXiv:2409.12122, 2024

    An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-math technical report: Toward mathematical expert model via se...

    2024 arXiv

  16. [24]

    Ovm, outcome-supervised value models for planning in mathematical reasoning

    Fei Yu, Anningzhe Gao, and Benyou Wang. Ovm, outcome-supervised value models for planning in mathematical reasoning. In Kevin Duh, Helena Gómez-Adorno, and Steven Bethard, editors,Findings of the Association for Computational Linguistics: NAACL 2024, Mexico City, Mexico, June ...

    2024

  17. [25]

    Free process rewards without process labels.arXiv preprint arXiv:2412.01981, 2024

    Lifan Yuan, Wendi Li, Huayu Chen, Ganqu Cui, Ning Ding, Kai Zhang, Bowen Zhou, Zhiyuan Liu, and Hao Peng. Free process rewards without process labels.arXiv preprint arXiv:2412.01981, 2024

    2024 arXiv

  18. [26]

    Generative verifiers: Reward modeling as next-token prediction.arXiv preprint arXiv:2408.15240, 2024

    Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction.arXiv preprint arXiv:2408.15240, 2024

    2024 arXiv

  19. [27]

    The lessons of developing process reward models in mathematical reasoning.arXiv preprint arXiv:2501.07301, 2025

    Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. The lessons of developing process reward models in mathematical reasoning.arXiv preprint arXiv:2501.07301, 2025. 10

    2025 arXiv

  20. [28]

    Limitations

    Chujie Zheng, Zhenru Zhang, Beichen Zhang, Runji Lin, Keming Lu, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. Processbench: Identifying process errors in mathematical reasoning.arXiv preprint arXiv:2412.06559, 2024. 11 A Derivations and Proofs The FreePRM loss functi...

    2024 arXiv

  21. [30]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals...

    2025

  22. [2024]

    URL:https://huggingface.co/Skywork

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.