Pith. sign in

REVIEW 4 cited by

Efficient Process Reward Model Training via Active Learning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2504.10559 v1 pith:R5NDLVTF submitted 2025-04-14 cs.LG cs.AI

Keyu Duan , Zichen Liu , Xin Mao , Tianyu Pang , Changyu Chen , Qiguang Chen , Michael Qizhe Shieh , Longxu Dou This is my paper
classification cs.LGcs.AI
keywords trainingactprmdataactiveannotationlearningmodelslabels
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Process Reward Models (PRMs) provide step-level supervision to large language models (LLMs), but scaling up training data annotation remains challenging for both humans and LLMs. To address this limitation, we propose an active learning approach, ActPRM, which proactively selects the most uncertain samples for training, substantially reducing labeling costs. During training, we use the PRM to estimate uncertainty after the forward pass, retaining only highly uncertain data. A capable yet costly reasoning model then labels this data. Then we compute the loss with respect to the labels and update the PRM's weights. We compare ActPRM vs. vanilla fine-tuning, on a pool-based active learning setting, demonstrating that ActPRM reduces 50% annotation, but achieving the comparable or even better performance. Beyond annotation efficiency, we further advance the actively trained PRM by filtering over 1M+ math reasoning trajectories with ActPRM, retaining 60% of the data. A subsequent training on this selected dataset yields a new state-of-the-art (SOTA) PRM on ProcessBench (75.0%) and PRMBench (65.5%) compared with same sized models.

Discussion (0). Sign in to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Sampling for Quality: Training-Free Reward-Guided LLM Decoding via Sequential Monte Carlo

    cs.LG 2026-04 unverdicted novelty 7.0 of 10

    Sequential Monte Carlo sampling from a reward-augmented sequence distribution improves LLM performance on HumanEval by up to 54.9% and MATH500 by up to 8.8%, outperforming standard sampling and GRPO.

  2. ProbeLLM: Automating Principled Diagnosis of LLM Failures

    cs.CL 2026-02 conditional novelty 6.0 of 10

    An automated LLM probing pipeline using hierarchical Monte Carlo Tree Search, tool-based verification, and failure clustering to discover structured failure modes beyond static benchmarks.

  3. When Self-Belief Misleads: Active Label Acquisition for Reinforcement Learning with Verifiable Rewards

    cs.LG 2026-05 unverdicted novelty 5.0 of 10

    RLAVR uses the Corrective Advantage Gap metric and CARE policy to actively acquire ground-truth labels for key samples, stabilizing RLVR training and boosting performance with limited annotation budgets.

  4. Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models

    cs.AI 2025-03 unverdicted novelty 5.0 of 10

    The paper unifies perspectives on Long CoT in reasoning LLMs by introducing a taxonomy, detailing characteristics of deep reasoning and reflection, and discussing emergence phenomena and future directions.

Pith tools