REVIEW 7 cited by
Process Reward Models for LLM Agents: Practical Framework and Directions
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We introduce Agent Process Reward Models (AgentPRM), a simple and scalable framework for training LLM agents to continually improve through interactions. AgentPRM follows a lightweight actor-critic paradigm, using Monte Carlo rollouts to compute reward targets and optimize policies. It requires minimal modifications to existing RLHF pipelines, making it easy to integrate at scale. Beyond AgentPRM, we propose InversePRM, which learns process rewards directly from demonstrations without explicit outcome supervision. We also explore key challenges and opportunities, including exploration, process reward shaping, and model-predictive reasoning. We evaluate on ALFWorld benchmark, show that small 3B models trained with AgentPRM and InversePRM outperform strong GPT-4o baselines, and analyze test-time scaling, reward hacking, and more. Our code is available at: https://github.com/sanjibanc/agent_prm.
Forward citations
Cited by 7 Pith papers
-
ARCO: Adaptive Rubrics with Co-Evolution for Multi-Step LLM-Based Agents
ARCO introduces a co-evolving rubric model with generation and scoring heads plus a trajectory decomposition constraint that improves exact-match scores on multi-hop QA tasks over outcome, rubric, and process reward b...
-
MASPRM: Multi-Agent System Process Reward Model
MASPRM trains a per-agent, per-message value head from outcome-only MCTS rollouts and uses it to guide step-level beam search and MCTS, improving exact match on GSM8K by up to +30.7 points over a greedy multi-agent pass.
-
Reinforcement Learning for Machine Learning Engineering Agents
RL-trained Qwen2.5-3B outperforms prompted Claude-3.5-Sonnet and GPT-4o on 12 MLEBench tasks by an average of 22% and 24%, using two targeted RL modifications.
-
Encouraging Good Processes Without the Need for Good Answers: Reinforcement Learning for LLM Agent Planning
Decoupling LLM-agent planning from summarization and rewarding tool-call completeness rather than final-answer correctness improves planning by 8-12% and end-to-end answers by 5-6% over end-to-end RL baselines.
-
Unleashing Embodied Task Planning Ability in LLMs via Reinforcement Learning
A 7B LLM trained with sparse completion rewards and a GRPO-style algorithm reaches state-of-the-art on ALFWorld and ScienceWorld.
-
A Diagnostic Framework for AI Agent Behavior
A two-layer diagnostic framework for AI agent behavior: distinguishing foundational computational substrate from behavioral modulation layer.
-
Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation
A taxonomy-driven survey arguing that reward design is the central mechanism shaping reliable LLM reasoning, with maps of reward paradigms, reward-hacking failure modes, and benchmark pitfalls.
Discussion (0). Sign in to comment.