Pith. sign in

REVIEW 23 cited by

GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2504.00891 v2 pith:V3Q3XLNR submitted 2025-04-01 cs.CL

Jian Zhao , Runze Liu , Kaiyan Zhang , Zhimu Zhou , Junqi Gao , Dong Li , Jiafei Lyu , Zhouyi Qian
show 3 more authors
Biqing Qi Xiu Li Bowen Zhou
This is my paper
classification cs.CL
keywords genprmprocessprmsllmsmodelmodelsreasoningcode
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Recent advancements in Large Language Models (LLMs) have shown that it is promising to utilize Process Reward Models (PRMs) as verifiers to enhance the performance of LLMs. However, current PRMs face three key challenges: (1) limited process supervision and generalization capabilities, (2) dependence on scalar value prediction without leveraging the generative abilities of LLMs, and (3) inability to scale the test-time compute of PRMs. In this work, we introduce GenPRM, a generative process reward model that performs explicit Chain-of-Thought (CoT) reasoning with code verification before providing judgment for each reasoning step. To obtain high-quality process supervision labels and rationale data, we propose Relative Progress Estimation (RPE) and a rationale synthesis framework that incorporates code verification. Experimental results on ProcessBench and several mathematical reasoning tasks show that GenPRM significantly outperforms prior PRMs with only 23K training data from MATH dataset. Through test-time scaling, a 1.5B GenPRM outperforms GPT-4o, and a 7B GenPRM surpasses Qwen2.5-Math-PRM-72B on ProcessBench. Additionally, GenPRM demonstrates strong abilities to serve as a critic model for policy model refinement. This work establishes a new paradigm for process supervision that bridges the gap between PRMs and critic models in LLMs. Our code, model, and data will be available in https://ryanliu112.github.io/GenPRM.

Discussion (0). Sign in to comment.

Forward citations

Cited by 23 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. KV-PRM: Efficient Process Reward Modeling via KV-Cache Transfer for Multi-Agent Test-Time Scaling

    cs.AI 2026-07 conditional novelty 7.0 of 10

    KV-PRM reuses generation KV caches plus a single verify token to score process rewards at O(L) cost, matching text-PRMs with up to 5000 imes fewer FLOPs on MATH/GSM8K/AIME.

  2. The Hidden Bias of Process Reward Models:PRISM for Rewarding the Right Reasoning

    cs.LG 2026-06 unverdicted novelty 7.0 of 10

    PRISM is a contrastive, policy-aware training framework for process reward models that reduces false positives by 22% on PRMBench and boosts downstream accuracy up to 33% in Best-of-N selection by learning reliable re...

  3. Not only where, But when: Temporal Scheduling for RLVR

    cs.LG 2026-05 unverdicted novelty 7.0 of 10

    Temporal scheduling of credit allocation criteria over RLVR training, using trajectory percentiles to target heterogeneous behaviors, yields more stable policy entropy and better reasoning benchmark results than stati...

  4. Unsupervised Process Reward Models

    cs.LG 2026-05 unverdicted novelty 7.0 of 10

    Unsupervised PRMs derived from LLM probabilities achieve up to 15% better error detection than LLM judges and match supervised PRMs in verification and RL tasks.

  5. Rewarding the Scientific Process: Process-Level Reward Modeling for Agentic Data Analysis

    cs.CL 2026-04 unverdicted novelty 7.0 of 10

    DataPRM is a new process reward model for data analysis agents that detects silent errors via environment interaction and ternary rewards, yielding 7-11% gains on benchmarks and further RL improvements.

  6. Rewarding the Scientific Process: Process-Level Reward Modeling for Agentic Data Analysis

    cs.CL 2026-04 unverdicted novelty 7.0 of 10

    DataPRM is an environment-aware generative process reward model that improves LLM data analysis agents by 7-11% on benchmarks via active verification and reflection-aware ternary rewards.

  7. Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learning

    cs.LG 2026-04 unverdicted novelty 7.0 of 10

    This survey introduces the Generate-Filter-Control-Replay (GFCR) taxonomy to structure rollout pipelines for RL-based post-training of reasoning LLMs.

  8. OpenClaw-RL: Train Any Agent Simply by Talking

    cs.CL 2026-03 unverdicted novelty 6.0 of 10

    OpenClaw-RL recovers evaluative and directive signals from next-state interactions to enable online RL training of agents across terminal, GUI, SWE, and tool environments via a server-client architecture and hybrid objective.

  9. Rethinking Reward Models for Multi-Domain Test-Time Scaling

    cs.AI 2025-10 conditional novelty 6.0 of 10

    Across 14 domains, a generative outcome reward model (gORM) outperforms both discriminative and generative process reward models for selecting best-of-N LLM reasoning chains.

  10. Beyond Correctness: Harmonizing Process and Outcome Rewards through RL Training

    cs.LG 2025-09 unverdicted novelty 6.0 of 10

    PROF curates RL training data via PRM-ORM consistency to improve both final-answer accuracy and intermediate reasoning quality while reducing reliance on strong process reward models.

  11. LLaVA-Critic-R1: Your Critic Model is Secretly a Strong Policy Model

    cs.CV 2025-08 conditional novelty 6.0 of 10

    RL training on preference-labeled critic data transforms a 7B vision-language model into both a stronger critic and a stronger generative policy, improving average benchmark accuracy by 5.7% and enabling self-critique...

  12. GM-PRM: A Generative Multimodal Process Reward Model for Multimodal Mathematical Reasoning

    cs.CL 2025-08 conditional novelty 6.0 of 10

    A generative multimodal process reward model that produces step-level critiques and corrections improves average math accuracy for six multimodal LLMs by 2.9 to 5.9 points under a refinement-based Best-of-N strategy.

  13. Stabilizing Knowledge, Promoting Reasoning: Dual-Token Constraints for RLVR

    cs.CL 2025-07 unverdicted novelty 6.0 of 10

    Archer introduces response-level entropy normalization and differentiated clipping/KL regularization in RLVR to encourage exploration on reasoning tokens while stabilizing knowledge tokens, yielding gains in pass@1 an...

  14. CoLD: Counterfactually-Guided Length Debiasing for Process Reward Models in Mathematical Reasoning

    cs.CL 2025-07 unverdicted novelty 6.0 of 10

    CoLD mitigates length bias in process reward models for mathematical reasoning via counterfactual guidance, length penalties, bias estimation, and joint training, improving step selection accuracy and conciseness on M...

  15. Test-Time Scaling for Small VLMs on Multilingual Visual MCQ

    cs.CL 2026-07 accept novelty 5.5 of 10

    On EXAMS-V, token budget and parseability dominate chain count, PRM-guided search, and selectors for small VLMs; the policy model itself yields the largest gain.

  16. Proxy Exploration and Reusable Guidance: A Modular LLM Post-Training Paradigm via Proxy-Guided Update Signals

    cs.LG 2026-07 conditional novelty 5.0 of 10

    Relative policy-improvement signals from a weak proxy model, after simple calibration, can be transferred to improve stronger primary LLMs without re-exploring on the primary.

  17. Test-Time Scaling in Multimodal Foundation Models: A Comprehensive Survey of Generation and Reasoning

    cs.CV 2026-06 unverdicted novelty 5.0 of 10

    A survey of test-time scaling for multimodal foundation models that introduces a three-way taxonomy of sampling, feedback, and search approaches along with applications and benchmarks.

  18. Trust Region On-Policy Distillation

    cs.LG 2026-05 unverdicted novelty 5.0 of 10

    TrOPD stabilizes on-policy distillation for LLMs with trust-region learning, outlier estimation, and off-policy guidance, outperforming prior OPD methods on reasoning and code benchmarks.

  19. Cut Your Losses! Learning to Prune Paths Early for Efficient Parallel Reasoning

    cs.CL 2026-04 unverdicted novelty 5.0 of 10

    STOP is a new learnable internal path-pruning technique that improves efficiency and accuracy of parallel reasoning in LRMs under fixed compute budgets.

  20. Cut Your Losses! Learning to Prune Paths Early for Efficient Parallel Reasoning

    cs.CL 2026-04 unverdicted novelty 5.0 of 10

    STOP, a learnable internal super-token pruner, is claimed to raise parallel-reasoning accuracy and efficiency on LRMs from 1.5B to 20B, e.g. AIME25 from 84% to ~90% under fixed compute.

  21. VRPRM: Process Reward Modeling via Visual Reasoning

    cs.LG 2025-08 conditional novelty 5.0 of 10

    VRPRM combines 3.6K CoT-PRM SFT data with 50K non-CoT PRM RL data to train a visual PRM that beats a 400K-data non-thinking PRM and boosts best-of-N accuracy.

  22. VRPRM: Process Reward Modeling via Visual Reasoning

    cs.LG 2025-08 unverdicted novelty 5.0 of 10

    VRPRM combines visual reasoning with a two-stage SFT-plus-RL strategy to deliver higher-quality process reward modeling using far less annotated data than prior non-thinking PRMs.

  23. Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle

    cs.CL 2025-09 conditional novelty 3.0 of 10

    A survey that maps reinforcement learning methods, datasets, benchmarks, and open-source tools across the full training lifecycle of large language models, focusing on verifiable-reward reasoning.

Pith tools