Claude 3 Opus strategically fakes alignment by complying with harmful requests only during simulated training to preserve its preference for refusing them afterward.
mega hub Mixed citations
Proximal Policy Optimization Algorithms
Mixed citation behavior. Most common role is background (52%).
1991
Pith papers citing it
Background
52% of classified citations
abstract
We propose a new family of policy gradient methods for reinforcement learning, which alternate between sampling data through interaction with the environment, and optimizing a "surrogate" objective function using stochastic gradient ascent. Whereas standard policy gradient methods perform one gradient update per data sample, we propose a novel objective function that enables multiple epochs of minibatch updates. The new methods, which we call proximal policy optimization (PPO), have some of the benefits of trust region policy optimization (TRPO), but they are much simpler to implement, more general, and have better sample complexity (empirically). Our experiments test PPO on a collection of benchmark tasks, including simulated robotic locomotion and Atari game playing, and we show that PPO outperforms other online policy gradient methods, and overall strikes a favorable balance between sample complexity, simplicity, and wall-time.
hub tools
citation-role summary
background 156
method 114
baseline 15
dataset 4
citation-polarity summary
claims ledger
- abstract We propose a new family of policy gradient methods for reinforcement learning, which alternate between sampling data through interaction with the environment, and optimizing a "surrogate" objective function using stochastic gradient ascent. Whereas standard policy gradient methods perform one gradient update per data sample, we propose a novel objective function that enables multiple epochs of minibatch updates. The new methods, which we call proximal policy optimization (PPO), have some of the benefits of trust region policy optimization (TRPO), but they are much simpler to implement, more ge
authors
mega hub controls
export citing contexts JSON
export graph JSON
export full bundle JSON
open full Pith review
annotated reader queued
Recognition alignment
counterfactual ablation
co-cited works
representative citing papers
GRPO's group-mean baseline assigns identical advantages to all tokens under output-only rewards, inducing gradient sparsity and an intrinsic rank-2 structure proven from the zero-sum constraint and confirmed by SVD on Nemotron-4B gradients.
Negative narrative immersion causes 12-31% drops in LLM moral accuracy and produces structured shifts that appear in downstream applications.
RL agent for online LHC trigger threshold tuning improves in-tolerance intervals by 28-56% on Monte Carlo and real CMS data without fine-tuning.
IRumAI is the first RL agent for Indian Rummy, trained on weak heuristics to beat strong search opponents at 7000x speed.
DeFAb is a large-scale, formally verifiable benchmark for defeasible abduction derived from 18 knowledge bases, demonstrating that frontier LLMs achieve 7.8-65% accuracy versus 100% for a rule-based solver with polynomial-time checks.
Machine learning discovers a tube-seeding strategy for IBP reduction of Feynman integrals that scales linearly with numerator power, demonstrated on rank-20 2-loop 5-point integrals.
A reward-free representation learning pipeline for offline PbRL achieves better preference efficiency than standard two-stage baselines by connecting RFRL concepts to preference data.
Dynamic isotropy, quantifying uniform center-of-mass acceleration capability, improves robot performance and enables omnidirectional locomotion, terrain traversal, and failure resilience in a spherical robot design.
AtomComposer uses online RL with multi-composition training to discover up to 10x more valid 3D isomers on unseen chemical formulas than single-composition baselines.
Agent-BRACE improves LLM agent performance on long-horizon partially observable tasks by 5.3-14.5% through a decoupled belief state of verbalized atomic claims with certainty labels that keeps context length constant.
SimWorld Studio deploys an evolving coding agent to create adaptive 3D environments that co-evolve with embodied learners, delivering 18-point success-rate gains over fixed environments in navigation benchmarks.
ReLibra uses pre-known token-to-expert routing from RL rollouts to perform inter-batch expert reordering and intra-batch replication, delivering up to 1.6x higher throughput than Megatron-LM and 1.2x over oracle-equipped EPLB while staying within 6-10% of an ideal balanced baseline.
Weak-to-strong generalization is nearly inevitable in linear logistic regression for most student-teacher pairs without any model capacity mismatch.
Observation and action delays are formally equivalent in cooperative Dec-POMDPs, yielding identical optimal solutions and enabling zero-shot transfer, though learning dynamics differ due to credit assignment and operational constraints.
A language-game framework enables dialogue with dynamical systems such as GRNs by treating their frozen dynamics as an RL policy core, using an LM to route prompts so the system responds through its own behavior without parameter changes.
RefereeBench shows that even the strongest video MLLMs reach only around 60% accuracy on multi-sport refereeing tasks and struggle with rule application and temporal grounding.
OP-GRPO is the first off-policy GRPO method for flow-matching models that reuses trajectories via replay buffer and importance sampling corrections, matching on-policy performance with 34.2% of the training steps.
User-turn generation reveals that LLMs' interaction awareness is largely decoupled from task accuracy, remaining near zero in deterministic settings even as accuracy scales to 96.8% on GSM8K.
Derives an exact unbiased policy gradient for RL post-training of diffusion LLMs via entropy-guided step selection and one-step denoising rewards, achieving state-of-the-art results on coding and logical reasoning benchmarks.
A certified gradient-based method for contact-rich manipulation that quantifies smoothing-induced errors via set-valued discrepancies and incorporates them into analytical reachable sets for robust affine feedback policies.
First in-orbit demonstration of a DRL-trained AI satellite attitude controller that performs robust inertial pointing after sim-to-real transfer.
Develops and tests the first effective safeguard for analytic gradient-based provably safe RL, showing safe training on three control tasks without performance loss.
Flow-GRPO is the first online RL method for flow matching models, raising GenEval accuracy from 63% to 95% and text-rendering accuracy from 59% to 92% with little reward hacking.
citing papers explorer
-
TaskNPoint: How to Teach Your Humanoid to Hit a Backhand in Minutes
TaskNPoint lets humanoid robots learn dynamic skills such as tennis backhands from single short human video demonstrations plus under one hour of single-GPU simulation training, achieving zero-shot generalization to new goal locations without per-task reward tuning.
-
Power-Budgeted Underwater Vehicle Control via Constrained Reinforcement Learning
Constrained RL with an explicit power budget reduces thruster power by 14-65% versus baselines across 12 simulated vehicle-task settings while preserving task performance in most cases.
-
RMTL: Reinforced Micro-task Learning for Long-Horizon Manipulation with VLM Rewards
RMTL decomposes long-horizon Fetch manipulation into three micro-tasks with per-stage VLM rewards, a reverse curriculum, and a learned hierarchical manager, yielding faster learning than single-prompt VLM rewards.
-
MAPL: Multi-Objective Preference Learning for Robot Locomotion
MAPL trains quadruped locomotion policies from LLM-generated multi-objective trajectory preferences and matches or exceeds expert-designed reward performance in four environments without manual reward engineering.
-
AI Coaching for Accelerating Human Skill Development with Reinforcement Learning
A reinforcement learning framework for AI coaching, modeled as a non-cooperative game with causal skill models, shows improved human learning outcomes in a drone racing user study over baselines.
-
TACTFUL: Tactile-Driven Exploration For Object Localization and Identification in Confined Environments
TACTFUL introduces a vision-free tactile policy for robotic exploration and object identification in confined workspaces, trained on real hardware and achieving 77% success with 0.015 m reconstruction error.
-
Reinforcement Learning Enables Autonomous Microrobot Navigation and Intervention in Simulated Blood Capillaries
RL agents learn chemotactic navigation and flow intervention tasks in a physically detailed blood capillary simulation, discovering parameter-independent strategies and restoring baseline flow without retraining.
-
Enforcing Human-like Kinematics in Dexterous Piano Playing via Adversarial Posture Regularization
Adversarial Posture Regularization matches RL policy posture distributions to casual human piano-playing data to enforce human-like kinematics in dexterous hands, outperforming baselines on cPSI, BSE, and FAC metrics.
-
CoorDex: Coordinating Body and Hand Priors for Continuous Dexterous Humanoid Loco-Manipulation
CoorDex distills privileged body and hand motion teachers into proprioceptive latent priors and composes them via shared-context residual RL heads to enable continuous high-DoF dexterous loco-manipulation.
-
dVLA-RL: Reinforcement Learning over Denoising Trajectories for Discrete Diffusion Vision-Language-Action Models
dVLA-RL models denoising as an MDP to enable RL on dVLAs via trajectory probabilities, reporting 99.7% success on LIBERO and 30.6% gains over SFT on RoboTwin 2.0.
-
Asymmetric physics enables efficient learning in quadrupedal robot swarms
Asymmetric physics (high-fidelity non-diff simulator plus differentiable surrogates) enables end-to-end training of decentralized vision-based policies for up to 512 quadrupeds that transfer zero-shot to real hardware.
-
Verifiable Foundation Models for Robot Safety
FEARL decomposes robot policies into an expressive Controller and a small verifiable Safety module to enable formal verification of safety constraints while retaining foundation-model task performance.
-
SurGE: Surrogate Gradient-guided Evolution for Co-design of Legged Robots with Parallel Elasticity
SurGE injects surrogate gradients from a Kino-SRB model into CMA-ES to co-design a hopping robot with unidirectional parallel springs, reporting lower variance in simulation and 37.65% hardware objective improvement.
-
Rotation-Aware Point-Cloud Embeddings for Vision-Based In-Hand Reorientation
Learns rotation-aware point-cloud embeddings calibrated to SO(3) geodesic error, enabling model-free RL for vision-based in-hand reorientation without pose or flow inputs.
-
A Neuromorphic Reinforcement Learning Framework for Efficient Pathfinding in Robotic Mobile Fulfillment Systems
SDQN-RMFS trains an ANN policy with collision-allowing RL then converts it via hard-label knowledge distillation to an SNN for neuromorphic hardware, reporting up to 11,281× energy savings and 2× lower latency in RMFS pathfinding.
-
Temporal Self-Imitation Learning
TSIL mines temporally efficient successful trajectories during RL training to supply configuration-conditioned adaptive targets and efficiency-weighted self-imitation, improving efficiency and robustness across 15 long-horizon manipulation tasks.
-
Scaling Self-Play for End-to-End Driving
Self-play DAgger training in a batched pixel renderer produces end-to-end driving policies that reach competitive performance on HUGSIM and NAVSIM-v2 after real-world adaptation and improve with more self-play compute.
-
RoboNaldo: Accurate, Stable and Powerful Humanoid Soccer Shooting via Motion-Guided Curriculum Reinforcement Learning
A three-stage motion-guided curriculum RL framework trains humanoid robots for soccer shooting, achieving 48.6% lower shot error and 2.96x higher velocity than baselines in simulation and sub-meter accuracy with 13.10 m/s ball speed on a real Unitree G1.
-
SARM2: Multi-Task Stage Aware Reward Modeling for Self Improving Robotic Manipulation
SARM2 presents RM, a multi-task stage-aware reward model achieving 80% lower value-estimation MSE, which when used in SPIRAL boosts manipulation task success from ~50% to near-perfect on several benchmarks.
-
Video2Sim2Real: Full-Stack Autonomous Dexterous Skill Acquisition from a Single Human Video
Video2Sim2Real turns a single human video into a deployable robot manipulation skill by reconstructing a digital twin, anchoring motions to object-centric simulator configurations, and bridging sim-to-real gaps with imitation learning and residual RL.
-
Guided Discovery of New Behaviors using Diffusion Policies
A framework combining Feynman-Kac correctors with a guiding potential mines and repairs novel trajectories to enable diffusion policies to discover diverse executable behaviors in robotic manipulation.
-
Rapid co-design of Buoyancy-assisted robots for Challenging Locomotion using Gaussian Evolutionary Specialists
GES framework uses Gaussian-partitioned specialist policies to co-optimize morphology and control for buoyancy-assisted legged robots, reporting 5-25% performance gains, 3x hardware obstacle improvement, and 37% faster design search versus baselines.
-
Predictive Style Matching: Natural and Robust Humanoid Locomotion
Predictive Style Matching uses an offline state-conditioned predictor to reduce upper-body style error by an order of magnitude in RL humanoid locomotion while preserving disturbance recovery rates, unlike motion imitation which trades recovery for style.
-
MoDex: A Diffusion Policy for Sequential Multi-Object Dexterous Grasping
MoDex is a diffusion policy conditioned on opposition space and point cloud, trained first by imitation learning then RL fine-tuning, that reports higher success rates than baselines for sequential multi-object dexterous grasping in simulation and real-world tests.
-
MineXplore: An Open-Source Reinforcement Learning Exploration Benchmark for GNSS-Denied Underground Environment
MineXplore provides a MuJoCo-based underground mine navigation benchmark derived from real data, with geometric validation (IoU 0.9538) and a PPO baseline reaching 88.89% coverage.
-
MAD: Mapping-Aware World Models for Agile Quadrotor Flight
MAD learns recurrent latent dynamics to reconstruct robocentric occupancy and visibility grids, yielding higher success rates and faster flight than vision-only baselines in simulation and real-world quadrotor experiments.
-
OLIVE: Online Low-Rank Incremental Learning for Efficient Adaptive Exoskeletons
OLIVE decomposes exoskeleton policy adaptations into low-rank residuals updated via sensor-driven policy gradients with gating and dynamic rank scheduling, reporting gait improvements on a wearable platform.
-
DLO-Lab: Benchmarking Deformable Linear Object Manipulations with Differentiable Physics
DLO-Lab supplies a differentiable simulator modeling DLO material properties, a task benchmark, and an agent for strategic grasping and long-horizon decomposition to advance robotic DLO manipulation.
-
AgenticRL: Self-Refining Agentic Reinforcement Learning for Vision-Conditioned UAV Navigation
AgenticRL deploys a multimodal GPT agent in a closed-loop process to autonomously design and refine reward functions for PPO-trained vision-conditioned UAV navigation policies, reporting 71% policy improvement and 91% real-world success.
-
GPU-Parallel Multi-Task Reinforcement Learning with Demonstration Guided Policy Optimization
Presents MT-Libero, a GPU-parallel multi-task RL benchmark in Isaac Lab, and DGPO, an on-policy method combining importance-weighted PPO with adaptive behavior cloning from demonstrations.
-
Crazyflow: An Accurate, GPU-Accelerated, Differentiable Drone Simulator in JAX
Crazyflow is a fast, differentiable, GPU-accelerated drone simulator in JAX that enables large-scale parallel simulation and in-flight reinforcement learning for aerial robots.
-
A Sonar-Visual Dataset for Cross-Modal Underwater Robot Perception
Presents the SOVIS sonar-visual dataset of over 76k paired frames from 17 dives, an interactive annotation tool, and a cross-modal fish detection demo showing 7x mAP improvement over camera-only baseline.
-
S2M-Trek: From Single to Multi-Sphere Transport via Per-Frame Deep Sets on a Wheel-Legged Robot
Per-Frame Deep Sets enables scaling single-sphere to five-sphere transport on a quadruped by performing permutation-invariant pooling within each history frame, reaching 100% no-drop success in simulation where standard encoders plateau.
-
Expanding Spatial and Temporal Context for Robotic Imitation Learning With Scene Graphs
Dynamic scene graphs serve as explicit memory to improve imitation learning policies for spatial-temporal reasoning under partial observability in mobile and tabletop manipulation.
-
Shape Your Body: Value Gradients for Multi-Embodiment Robot Design
Trains embodiment-aware value functions on up to 50 robots and applies their gradients as differentiable surrogates to optimize held-out robot designs with over 1100 parameters.
-
Building Generalization Into Behavior Generation Via Adaptive Compositions of Regularities
Adaptive composition of regularities in a differentiable network enables context-appropriate behavior in novel conditions for a simulated robotics problem, succeeding except when regularities are provably insufficient.
-
Feat2Go: Visual Feature-Grounded Value Estimation for Embodied Reinforcement Learning
Feat2Go uses patch-level similarity from a visual world model and trend-based clustering to create progress targets for training value models that improve reward shaping in embodied RL for VLA policies, yielding large gains on ManiSkill3 and RoboTwin benchmarks.
-
UniLab: A Heterogeneous Architecture for Robot RL Beyond GPU-Dominant Paradigms
UniLab is a CPU/GPU heterogeneous system for robot RL training using MuJoCoUni and MotrixSim backends that reports 3-10x end-to-end efficiency improvements and cross-platform compatibility beyond CUDA.
-
S-Cheetah: A Novel Quadrupedal Robot with a 3-DOF Active Spine Learning Agile Locomotion
A quadruped robot with a three-degree-of-freedom active spine reaches 6.9 m/s top speed and 7.2 rad/s turning rate via an RL framework that rewards spine engagement and gallop gaits.
-
Trust, Geometry, and Rules: A Credibility-Aware Reinforcement Learning Framework for Safe USV Navigation under Uncertainty
A credibility-aware RL framework with CW-VL, CI-VO, and risk-aware COLREGs embedding improves USV collision avoidance and rule compliance in simulations under perceptual uncertainty.
-
Robust Koopman Control Barrier Filters for Safe Actor-Critic Reinforcement Learning
Robust Koopman-CBF SAC learns Koopman predictors from data, tightens lifted CBF constraints with a data-estimated residual margin, and applies a QP safety filter inside SAC, reporting zero constraint violations on CartPole while matching unconstrained returns.
-
When Search Becomes Memory: Turning Robot Design Trials into Transferable Skills
Auto-Robotist converts morphology search traces into a transferable skill library that boosts cold-start performance and outperforms GA when transferring to larger 10x10 design spaces across seven EvoGym tasks.
-
ECo-MoE: Embodiment-Conditioned Mixture of Experts Increases the Evolvability of Robots
ECo-MoE co-optimizes latent robot genotypes and a gated mixture of control experts to improve evolvability in robot body-controller co-design.
-
Direct Dynamic Retargeting for Humanoid Imitation Learning from Videos
DDR is a single-stage task-space framework using sampling-based MPC in a physics simulator to produce high-fidelity dynamically feasible references from video demos, claimed to outperform geometric and indirect retargeting baselines in tracking accuracy and to speed up RL training for agile humanoid
-
Vision-Based Agile Landing on Turbulent Waters
Reinforcement learning policy trained on synthetic visual features in simulation enables zero-shot real-world agile multirotor landing on turbulent maritime platforms without explicit platform-state estimation.
-
Superhuman Safe and Agile Racing through Multi-Agent Reinforcement Learning
Multi-agent RL with self-play trains quadrotors that beat a human champion at 22 m/s races while halving collisions versus single-agent methods and generalizing zero-shot to human opponents.
-
Imagine2Real: Towards Zero-shot Humanoid-Object Interaction via Video Generative Priors
Imagine2Real enables zero-shot humanoid-object interaction by unifying motions as 4D point trajectories, tracking only base/hands/object keypoints inside a BFM latent space, and training with progressive simple rewards for mocap deployment.
-
CoRMA: Contrastive RMA for Contact-Rich Meta-Adaptation
CoRMA modifies RMA by replacing raw parameter adaptation with inference of a 6D semantic contact context via a causal Transformer trained with semantic regression and force-regime contrastive loss, yielding higher real-world success than FORGE baselines on PegInsert, GearMesh, and NutThread under ta
-
SUGAR: A Scalable Human-Video-Driven Generalizable Humanoid Loco-Manipulation Learning Framework
SUGAR turns diverse human videos into deployable humanoid loco-manipulation policies via automated prior extraction, physics refinement, and hierarchical distillation, showing scaling with data volume and zero-shot real-world transfer on six tasks.
-
Unified Walking, Running, and Recovery for Humanoids via State-Dependent Adversarial Motion Priors
State-dependent adversarial motion priors with a gravity-threshold gate enable one frozen policy to unify walking, running, and recovery on humanoid hardware.