REVIEW 7 cited by
From Novice to Expert: LLM Agent Policy Optimization via Step-wise Reinforcement Learning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
From Novice to Expert: LLM Agent Policy Optimization via Step-wise Reinforcement Learning
read the original abstract
The outstanding capabilities of large language models (LLMs) render them a crucial component in various autonomous agent systems. While traditional methods depend on the inherent knowledge of LLMs without fine-tuning, more recent approaches have shifted toward the reinforcement learning strategy to further enhance agents' ability to solve complex interactive tasks with environments and tools. However, previous approaches are constrained by the sparse reward issue, where existing datasets solely provide a final scalar reward for each multi-step reasoning chain, potentially leading to ineffectiveness and inefficiency in policy learning. In this paper, we introduce StepAgent, which utilizes step-wise reward to optimize the agent's reinforcement learning process. Inheriting the spirit of novice-to-expert theory, we first compare the actions of the expert and the agent to automatically generate intermediate rewards for fine-grained optimization. Additionally, we propose implicit-reward and inverse reinforcement learning techniques to facilitate agent reflection and policy adjustment. Further theoretical analysis demonstrates that the action distribution of the agent can converge toward the expert action distribution over multiple training cycles. Experimental results across various datasets indicate that StepAgent outperforms existing baseline methods.
Forward citations
Cited by 7 Pith papers
-
TRACER: Verifiable Generative Provenance for Multimodal Tool-Using Agents
TRACER attaches verifiable sentence-level provenance records to multimodal agent outputs using tool-turn alignment and semantic relations, yielding 78.23% answer accuracy and fewer tool calls than baselines on TRACE-Bench.
-
SPS: Steering Probability Squeezing for Better Exploration in Reinforcement Learning for Large Language Models
SPS interleaves RL and IRL to counteract probability squeezing in LLM reasoning trajectories, improving Pass@k on five benchmarks while identifying an empirical upper bound on multi-sample performance.
-
RSPO: Reward-Swap Policy Optimization for Multi-Turn LLM Agents
Reward-Swap Policy Optimization uses dense-reward exploration to generate diverse trajectories that then train the final policy strictly under true outcome rewards, lifting multi-turn agent performance.
-
Process Reward Informed Tree Rollout for Effective Multi-Turn RL
Using process feedback to guide which partial trajectories to branch on, PATR replaces uniform independent rollouts with adaptive tree rollouts for multi-turn agent RL, reporting +5.0 points on SWE-Bench and +9.3 on F...
-
From Reasoning to Agentic: Credit Assignment in Reinforcement Learning for Large Language Models
A survey of credit assignment techniques in LLM reinforcement learning that distinguishes maturing methods for reasoning from new approaches needed for agentic settings and provides supporting resources.
-
Harnessing Uncertainty: Entropy-Modulated Policy Gradients for Long-Horizon LLM Agents
EMPG re-weights policy-gradient updates by step-level token entropy, amplifying confident correct actions and muting uncertain ones, and adds a future-clarity bonus.
-
ODYSSE: Episode-wise Policy Optimization for Personalized Agentic Reasoning
An episode-level variant of GRPO (ESPO) improves personalized GUI-agent reasoning on the 102-episode SmartSpot benchmark, outperforming step-wise and outcome-only training baselines.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.