REVIEW 21 cited by
Reward Design with Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Reward design in reinforcement learning (RL) is challenging since specifying human notions of desired behavior may be difficult via reward functions or require many expert demonstrations. Can we instead cheaply design rewards using a natural language interface? This paper explores how to simplify reward design by prompting a large language model (LLM) such as GPT-3 as a proxy reward function, where the user provides a textual prompt containing a few examples (few-shot) or a description (zero-shot) of the desired behavior. Our approach leverages this proxy reward function in an RL framework. Specifically, users specify a prompt once at the beginning of training. During training, the LLM evaluates an RL agent's behavior against the desired behavior described by the prompt and outputs a corresponding reward signal. The RL agent then uses this reward to update its behavior. We evaluate whether our approach can train agents aligned with user objectives in the Ultimatum Game, matrix games, and the DealOrNoDeal negotiation task. In all three tasks, we show that RL agents trained with our framework are well-aligned with the user's objectives and outperform RL agents trained with reward functions learned via supervised learning
Forward citations
Cited by 21 Pith papers
-
Sign-SZPO: Provable Preference-based Reinforcement Learning with an Unknown Link Function
ZSPO provably converges to a stationary policy using only the sign of preference feedback, without knowing the link function between preferences and rewards.
-
Computational models of pragmatic reasoning with flexible generation of meaning and expression alternatives
Combining language-model generation with rule-based selection reproduces several pragmatic phenomena, but the language models only worked reliably as idea generators, not as judges of formal linguistic properties.
-
TAPAS: Throughput-adaptive Perception for Autonomous Systems
An RL-based scheduler chooses CPU/GPU/DLA clusters for each perception model to meet scene-derived frame-rate targets, reporting large energy savings, though the frame-rate targets are defined by a hand-tuned entropy mapping.
-
Debate2Create: Robot Co-design via Multi-Agent LLM Debate
A structured multi-agent LLM debate grounded in simulation is proposed for co-designing robot morphology and reward, but the body reports only a single Ant experiment.
-
RoboInspector: Unveiling the Unreliability of Policy Code for LLM-enabled Robotic Manipulation
LLM-generated robot policy code is unreliable, with failures clustering into four behavior types that grow with task complexity and shrink with instruction detail; a failure-feedback retry improves success up to 35%.
-
GPLight+: A Genetic Programming Method for Learning Symmetric Traffic Signal Control Policy
Enforcing symmetry in GP-evolved phase urgency functions, by sharing a single subtree between the two turn movements and summing their urgencies, improves traffic signal control performance on 5 of 6 benchmark datasets.
-
FOUNDER: Grounding Foundation Models in World Models for Open-Ended Embodied Decision Making
FOUNDER maps foundation-model embeddings of text or video prompts into world-model goal states and rewards policies by predicted temporal distance to those goals, improving reward-free multi-task offline control.
-
Agentic Episodic Control
AEC couples an LLM semantic encoder, a graph working memory, and a critical-state gate to make episodic control in text-based RL more sample-efficient than standard RL baselines.
-
LLM-Guided Reinforcement Learning: Addressing Training Bottlenecks through Policy Modulation
An LLM identifies critical states, suggests corrective actions, and assigns shaped rewards to refine an existing RL policy, beating several baselines in Pong and MuJoCo.
-
Where You Go is Who You Are: Behavioral Theory-Guided LLMs for Inverse Reinforcement Learning
SILIC uses LLM-guided inverse reinforcement learning and Theory of Planned Behavior chain reasoning to infer age, gender, income, and employment from travel trajectories, reportedly beating SVM, XGBoost, CatBoost, and...
-
InfiMed-ORBIT: Aligning LLMs on Open-Ended Complex Tasks via Rubric-Based Incremental Training
Rubric-based incremental RL with LLM-generated case-specific checklists lifts Qwen3-4B's HealthBench-Hard score from 7.0 to 27.5 with 2k samples, and improves InfoBench instruction-following from 42.0 to 82.9.
-
Efficient Onboard Vision-Language Inference in UAV-Enabled Low-Altitude Economy Networks via LLM-Enhanced Optimization
A hierarchical ARPO+LLaRA framework that jointly sets image resolution, transmit power, and UAV trajectory reduces simulated latency for onboard VLM inference in low-altitude economy networks.
-
Reward Evolution with Graph-of-Thoughts: A Bi-Level Language Model Framework for Reinforcement Learning
RE-GoT combines graph-of-thoughts planning in LLMs with VLM feedback from rollout videos to automatically write and refine RL reward functions, beating prior LLM-based reward design on RoboGen and ManiSkill2.
-
ACTLLM: Action Consistency Tuned Large Language Model
ACTLLM trains an LLM to jointly produce structured scene descriptions and actions, and reports improved compositional and zero-shot generalization on CLIPORT and VIMA.
-
Speculative Reward Model Boosts Decision Making Ability of LLMs Cost-Effectively
A plug-and-play external reward model with speculative rejection sampling cuts tree-search cost for LLM decision-making to about 1/10 while keeping or slightly improving accuracy on math, planning, and financial reaso...
-
Divide-Fuse-Conquer: Eliciting "Aha Moments" in Multi-Scenario Games
A group, fuse, and retrain recipe for multi-game reinforcement learning lets a 32B model reach near-Claude3.5 performance on several TextArena games, though the headline score is internally inconsistent.
-
Toward Edge General Intelligence with Agentic AI and Agentification: Concepts, Technologies, and Future Directions
A survey that organizes agentic AI for 6G edge networks into four pillars, compactness, efficiency, knowledge and reasoning, and migration, and illustrates them with prior case studies.
-
Edge Agentic AI Framework for Autonomous Network Optimisation in O-RAN
A simulated edge agentic AI framework with LSTM traffic prediction and tiered Tx power control reports zero network outages in high-stress 5G scenarios.
-
Towards Reliable, Uncertainty-Aware Alignment
Variance-aware RLHF adds a variance-weighted KL penalty to PPO and reduces reward variance and the risk of underperforming the reference policy in the paper's experiments.
-
Prompt Informed Reinforcement Learning for Visual Coverage Path Planning
Adding GPT-3.5 semantic recommendations as an auxiliary reward term to PPO improves visual coverage and reduces redundancy for simulated aerial coverage path planning, according to reported experiments.
-
An evaluation of LLMs for generating movie reviews: GPT-4o, Gemini-2.0 and DeepSeek-V3
LLMs can produce fluent movie reviews that readers often mistake for human-written ones, but the models differ in emotional balance and depth.
Discussion (0). Sign in to comment.