MPC-Injection biases off-policy RL locomotion policies toward controller-induced behavior basins by injecting MPC transitions into the replay buffer.
hub
Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
29 Pith papers cite this work. Polarity classification is still indexing.
29
Pith papers citing it
abstract
We propose a general and model-free approach for Reinforcement Learning (RL) on real robotics with sparse rewards. We build upon the Deep Deterministic Policy Gradient (DDPG) algorithm to use demonstrations. Both demonstrations and actual interactions are used to fill a replay buffer and the sampling ratio between demonstrations and transitions is automatically tuned via a prioritized replay mechanism. Typically, carefully engineered shaping rewards are required to enable the agents to efficiently explore on high dimensional control problems such as robotics. They are also required for model-based acceleration methods relying on local solvers such as iLQG (e.g. Guided Policy Search and Normalized Advantage Function). The demonstrations replace the need for carefully engineered rewards, and reduce the exploration problem encountered by classical RL approaches in these domains. Demonstrations are collected by a robot kinesthetically force-controlled by a human demonstrator. Results on four simulated insertion tasks show that DDPG from demonstrations out-performs DDPG, and does not require engineered rewards. Finally, we demonstrate the method on a real robotics task consisting of inserting a clip (flexible object) into a rigid object.
hub tools
citation-role summary
background 2
citation-polarity summary
roles
background 2polarities
background 2representative citing papers
WireCraft is a new configurable simulation benchmark for industrial DLO manipulation with three task families, dual physics models, and shared evaluation of RL, IL, and VLA policies showing high success under privileged state but bottlenecks for vision-based methods.
IWR improves CRL sample efficiency and performance in interaction-rich manipulation by interaction-aware resampling that preserves mode boundaries, yielding 19.8% average gains and a real-world air-hockey agent.
EXPO stabilizes online RL for expressive policies by training a base policy with imitation and using a lightweight Gaussian edit policy to select higher-value actions on the fly for sampling and TD backups.
ActGuide-RL uses human action data as plan-style guidance in mixed-policy RL to overcome exploration barriers in LLM agents, matching SFT+RL performance on search benchmarks without cold-start training.
A new meta-training algorithm, ELO, combines a failure-aware resume buffer with progressive expert supervision; its best learned optimizer, ELO-Celo2, outperforms AdamW on ImageNet and GPT-2 and matches Muon on language modeling.
RLVR training teaches a 30B LLM to strategically explore a multi-buyer market and extract 70% of available surplus, outperforming frontier models up to 1T parameters in concurrent negotiation.
AutoSERL achieves strong performance on six real-world robot manipulation tasks using RL guided by a single demonstration via sliding-window intervention, safety recovery, and automatic termination.
PACT uses demo-trained progress localization plus intervention preference pairs to correct inflated Bellman targets and align the actor, raising average real-robot success by 24.5% over HIL-SERL.
Demo2Reward optimizes VLM reward model language instructions at test time from a few demonstrations to reduce false positives and enable policy learning in simulated and real robotic tasks without manual reward design.
ROAD formulates data mixing as a bi-level optimization problem solved via multi-armed bandit to adaptively balance offline priors and online updates in RL.
SOPE dynamically controls offline training length in online RL using actor-aligned OPE on validation data to stop when benefits saturate, achieving up to 45.6% better performance and 22x less computation on Minari tasks.
VLA-RL applies online RL to pretrained VLAs, yielding a 4.5% gain over strong baselines on 40 LIBERO manipulation tasks and matching commercial models like π₀-FAST.
DPPO fine-tunes diffusion policies via policy gradients and outperforms prior RL approaches for diffusion policies and PG-tuned alternatives on robot benchmarks while enabling stable training and hardware deployment.
DD-SRad is a new RL constraint technique that adapts per-actuator radii dynamically to achieve zero violations and unconstrained-level task performance on heterogeneous robotic joints.
Sparse directional VLA guidance jump-starts PPO for long-horizon and poorly rewarded manipulation, improving sample efficiency over PPO and distillation while enabling zero-shot real-robot transfer.
PriPG-RL trains RL policies for POMDPs by distilling knowledge from a privileged anytime-feasible MPC planner into a P2P-SAC policy, improving sample efficiency and performance in partially observable robotic navigation.
FORCE is a 3-stage RL fine-tuning method for VLA models that stabilizes Q-function via on-policy warm-up and filters high-value actions for updates, claiming 79% success rate gains and 32.5% faster training without human intervention.
L-SDPPO optimizes a spiking diffusion policy with RL and adds SDLI to handle microgravity dynamics, reporting higher success rates and lower energy use than prior methods on five intra-vehicular tasks.
Dual-Window Smoothing uses an execution window for deterministic smoothness and a value window to correct critic bias, plus a first-order temporal regularizer, to achieve smoother RL control than explicit chunking or standard baselines.
LLM-TALE steers RL exploration using LLM-generated plans at task and affordance levels with online suboptimality correction, improving sample efficiency and success rates on pick-and-place tasks without human supervision.
Unsupervised behavioral mode discovery combined with mutual information rewards enables RL fine-tuning of multimodal generative policies that achieves higher success rates without losing action diversity.
XQCfD accelerates actor-critic RL by using prior data, pretrained policies, and stationary architectures to achieve state-of-the-art results on Adroit, Robomimic, and MimicGen manipulation benchmarks with low update-to-data ratios.
Soft-DPG uses Gaussian smoothing on the Bellman equation to derive a well-defined policy gradient without relying on critic action derivatives, yielding competitive performance on dense-reward tasks and gains on discretized-reward variants.
citing papers explorer
-
MPC-Injection: Biasing Off-Policy Locomotion RL Toward Controller-Induced Behavior Basins
MPC-Injection biases off-policy RL locomotion policies toward controller-induced behavior basins by injecting MPC transitions into the replay buffer.
-
WireCraft: A Simulation Benchmark for Industrial DLO Manipulation
WireCraft is a new configurable simulation benchmark for industrial DLO manipulation with three task families, dual physics models, and shared evaluation of RL, IL, and VLA policies showing high success under privileged state but bottlenecks for vision-based methods.
-
Learning Object Manipulation from Scratch via Contrastive Interaction
IWR improves CRL sample efficiency and performance in interaction-rich manipulation by interaction-aware resampling that preserves mode boundaries, yielding 19.8% average gains and a real-world air-hockey agent.
-
EXPO: Stable Reinforcement Learning with Expressive Policies
EXPO stabilizes online RL for expressive policies by training a base policy with imitation and using a lightweight Gaussian edit policy to select higher-value actions on the fly for sampling and TD backups.
-
Learning Agentic Policy from Action Guidance
ActGuide-RL uses human action data as plan-style guidance in mixed-policy RL to overcome exploration barriers in LLM agents, matching SFT+RL performance on search benchmarks without cold-start training.
-
Efficient Long-Horizon Learning for Learned Optimization
A new meta-training algorithm, ELO, combines a failure-aware resume buffer with progressive expert supervision; its best learned optimizer, ELO-Celo2, outperforms AdamW on ImageNet and GPT-2 and matches Muon on language modeling.
-
Strategic Bargaining in Multi-Buyer Markets: Reinforcement Learning from Verifiable Rewards for LLM Negotiations
RLVR training teaches a 30B LLM to strategically explore a multi-buyer market and extract 70% of available surplus, outperforming frontier models up to 1T parameters in concurrent negotiation.
-
One Demonstration Is Enough for Real-World Robotic Reinforcement Learning
AutoSERL achieves strong performance on six real-world robot manipulation tasks using RL guided by a single demonstration via sliding-window intervention, safety recovery, and automatic termination.
-
Preference-Calibrated Human-in-the-Loop Reinforcement Learning for Robotic Manipulation
PACT uses demo-trained progress localization plus intervention preference pairs to correct inflated Bellman targets and align the actor, raising average real-robot success by 24.5% over HIL-SERL.
-
From Demonstrations to Rewards: Test-Time Prompt Optimization for VLM Reward Models
Demo2Reward optimizes VLM reward model language instructions at test time from a few demonstrations to reduce false positives and enable policy learning in simulated and real robotic tasks without manual reward design.
-
ROAD: Adaptive Data Mixing for Offline-to-Online Reinforcement Learning via Bi-Level Optimization
ROAD formulates data mixing as a bi-level optimization problem solved via multi-armed bandit to adaptively balance offline priors and online updates in RL.
-
SOPE: Stabilizing Off-Policy Evaluation for Online RL with Prior Data
SOPE dynamically controls offline training length in online RL using actor-aligned OPE on validation data to stop when benefits saturate, achieving up to 45.6% better performance and 22x less computation on Minari tasks.
-
VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learning
VLA-RL applies online RL to pretrained VLAs, yielding a 4.5% gain over strong baselines on 40 LIBERO manipulation tasks and matching commercial models like π₀-FAST.
-
Diffusion Policy Policy Optimization
DPPO fine-tunes diffusion policies via policy gradients and outperforms prior RL approaches for diffusion policies and PG-tuned alternatives on robot benchmarks while enabling stable training and hardware deployment.
-
Constraint-Enhanced Reinforcement Learning Based on Dynamic Decoupled Spherical Radial Squashing
DD-SRad is a new RL constraint technique that adapts per-actuator radii dynamically to achieve zero violations and unconstrained-level task performance on heterogeneous robotic joints.
-
Vision-Language-Action Jump-Starting for Reinforcement Learning Robotic Agents
Sparse directional VLA guidance jump-starts PPO for long-horizon and poorly rewarded manipulation, improving sample efficiency over PPO and distillation while enabling zero-shot real-robot transfer.
-
PriPG-RL: Privileged Planner-Guided Reinforcement Learning for Partially Observable Systems with Anytime-Feasible MPC
PriPG-RL trains RL policies for POMDPs by distilling knowledge from a privileged anytime-feasible MPC planner into a P2P-SAC policy, improving sample efficiency and performance in partially observable robotic navigation.
-
FORCE: Efficient VLA Reinforcement Fine-Tuning via Value-Calibrated Warm-up and Self-Distillation
FORCE is a 3-stage RL fine-tuning method for VLA models that stabilizes Q-function via on-policy warm-up and filters high-value actions for updates, claiming 79% success rate gains and 32.5% faster training without human intervention.
-
L-SDPPO: Policy Optimization of Spiking Diffusion Policy for Intra-vehicular Robotic Manipulation
L-SDPPO optimizes a spiking diffusion policy with RL and adds SDLI to handle microgravity dynamics, reporting higher success rates and lower energy use than prior methods on five intra-vehicular tasks.
-
Implicit Action Chunking for Smooth Continuous Control
Dual-Window Smoothing uses an execution window for deterministic smoothness and a value window to correct critic bias, plus a first-order temporal regularizer, to achieve smoother RL control than explicit chunking or standard baselines.
-
LLM-Guided Task- and Affordance-Level Exploration in Reinforcement Learning
LLM-TALE steers RL exploration using LLM-generated plans at task and affordance levels with online suboptimality correction, improving sample efficiency and success rates on pick-and-place tasks without human supervision.
-
Behavioral Mode Discovery for Fine-tuning Multimodal Generative Policies
Unsupervised behavioral mode discovery combined with mutual information rewards enables RL fine-tuning of multimodal generative policies that achieves higher success rates without losing action diversity.
-
XQCfD: Accelerating Fast Actor-Critic Algorithms with Prior Data and Prior Policies
XQCfD accelerates actor-critic RL by using prior data, pretrained policies, and stationary architectures to achieve state-of-the-art results on Adroit, Robomimic, and MimicGen manipulation benchmarks with low update-to-data ratios.
-
Soft Deterministic Policy Gradient with Gaussian Smoothing
Soft-DPG uses Gaussian smoothing on the Bellman equation to derive a well-defined policy gradient without relying on critic action derivatives, yielding competitive performance on dense-reward tasks and gains on discretized-reward variants.
-
Sample-efficient Transfer Reinforcement Learning via Adaptive Reward Shaping and Policy-Ratio Reweighting Strategy
Introduces a safe transfer RL framework for highway lane changing with adaptive intervention, teacher-guided reward shaping, and weighted optimization, claiming over 52% safety and 5% efficiency gains over baselines.
-
Quantifying the Energy Floor: Direct Measurement and Replay Buffer Bias in SAC-Based HVAC Control on sbsim
The energy floor for SAC HVAC control on sbsim is $35.51/day, with replay buffer initialization from schedule policies causing 96% of the observed 4.7% suboptimality gap.
-
Trust Region Q Adjoint Matching
TRQAM adds a trust region to QAM by optimizing λ in SOC dynamics to achieve closed-form control of path-space KL, yielding 68% success rate on 50 OGBench tasks versus 46% for the strongest baseline.
-
On Multi-Agent Learning in Team Sports Games
Describes a hierarchical RL method for multi-agent learning in team sports games aiming for human-like agents, reporting preliminary results that show promise.
- Stable GFlowNets with TV Monitoring and Probabilistic Guarantees