REVIEW 8 cited by
Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Recent months have seen the emergence of a powerful new trend in which large language models (LLMs) are augmented to become autonomous language agents capable of performing objective oriented multi-step tasks on their own, rather than merely responding to queries from human users. Most existing language agents, however, are not optimized using environment-specific rewards. Although some agents enable iterative refinement through verbal feedback, they do not reason and plan in ways that are compatible with gradient-based learning from rewards. This paper introduces a principled framework for reinforcing large language agents by learning a retrospective model, which automatically tunes the language agent prompts from environment feedback through policy gradient. Specifically, our proposed agent architecture learns from rewards across multiple environments and tasks, for fine-tuning a pre-trained language model which refines the language agent prompt by summarizing the root cause of prior failed attempts and proposing action plans. Experimental results on various tasks demonstrate that the language agents improve over time and that our approach considerably outperforms baselines that do not properly leverage gradients from the environment. This demonstrates that using policy gradient optimization to improve language agents, for which we believe our work is one of the first, seems promising and can be applied to optimize other models in the agent architecture to enhance agent performances over time.
Forward citations
Cited by 8 Pith papers
-
S2T-RLHF: Hierarchical Credit Assignment for Stable Preference-Based RLHF
S2T-RLHF splits each response-level RLHF reward into sentence shares and then token shares, via bargaining and Dirichlet weighting, yielding steadier training with competitive preference alignment.
-
Tailored Conversations beyond LLMs: A RL-Based Dialogue Manager
A hierarchical RL and meta-learning dialogue manager conditions an LLM for motivational interviewing and reports higher reward than a prompted LLM baseline in a simulated environment.
-
Model Performance-Guided Evaluation Data Selection for Effective Prompt Optimization
An evaluation-set selection method that adds real-time model feedback to semantic sampling improves the accuracy and stability of three prompt optimization methods on two datasets.
-
Refining Answer Distributions for Improved Large Language Model Reasoning
RAD iteratively refines a distribution over answers by marginalizing over refinement samples, improving accuracy on six arithmetic benchmarks over self-consistency and hint-based prompting.
-
Xolver: Multi-Agent Reasoning with Holistic Experience Learning Just Like an Olympiad Team
A training-free multi-agent framework with episodic and shared memory reports new best results on GSM8K, AIME 2024/2025, Math-500, and LiveCodeBench.
-
ReflectEvo: Improving Meta Introspection of Small LLMs by Learning Self-Reflection
ReflectEvo shows that small language models can improve their reasoning by fine-tuning on their own self-generated reflections, but the headline BIG-bench gains depend on oracle feedback and an unclear data split.
-
A Survey of Foundation Model-Powered Recommender Systems: From Feature-Based, Generative to Agentic Paradigms
A survey that organizes foundation-model recommender systems into feature-based, generative, and agentic paradigms and reviews tasks, empirical results, and open challenges.
-
Large language models for artificial general intelligence (AGI): A survey of foundational principles and approaches
This survey argues that embodiment, symbol grounding, causality, and memory are the foundational principles needed to make large language models achieve artificial general intelligence.
Discussion (0). Continue with ORCID to comment.