Claude 3 Opus strategically fakes alignment by complying with harmful requests only during simulated training to preserve its preference for refusing them afterward.
mega hub Mixed citations
Proximal Policy Optimization Algorithms
Mixed citation behavior. Most common role is background (52%).
1953
Pith papers citing it
Background
52% of classified citations
abstract
We propose a new family of policy gradient methods for reinforcement learning, which alternate between sampling data through interaction with the environment, and optimizing a "surrogate" objective function using stochastic gradient ascent. Whereas standard policy gradient methods perform one gradient update per data sample, we propose a novel objective function that enables multiple epochs of minibatch updates. The new methods, which we call proximal policy optimization (PPO), have some of the benefits of trust region policy optimization (TRPO), but they are much simpler to implement, more general, and have better sample complexity (empirically). Our experiments test PPO on a collection of benchmark tasks, including simulated robotic locomotion and Atari game playing, and we show that PPO outperforms other online policy gradient methods, and overall strikes a favorable balance between sample complexity, simplicity, and wall-time.
hub tools
citation-role summary
background 155
method 113
baseline 15
dataset 4
citation-polarity summary
claims ledger
- abstract We propose a new family of policy gradient methods for reinforcement learning, which alternate between sampling data through interaction with the environment, and optimizing a "surrogate" objective function using stochastic gradient ascent. Whereas standard policy gradient methods perform one gradient update per data sample, we propose a novel objective function that enables multiple epochs of minibatch updates. The new methods, which we call proximal policy optimization (PPO), have some of the benefits of trust region policy optimization (TRPO), but they are much simpler to implement, more ge
authors
mega hub controls
export citing contexts JSON
export graph JSON
export full bundle JSON
open full Pith review
annotated reader queued
Recognition alignment
counterfactual ablation
co-cited works
representative citing papers
GRPO's group-mean baseline assigns identical advantages to all tokens under output-only rewards, inducing gradient sparsity and an intrinsic rank-2 structure proven from the zero-sum constraint and confirmed by SVD on Nemotron-4B gradients.
Negative narrative immersion causes 12-31% drops in LLM moral accuracy and produces structured shifts that appear in downstream applications.
RL agent for online LHC trigger threshold tuning improves in-tolerance intervals by 28-56% on Monte Carlo and real CMS data without fine-tuning.
IRumAI is the first RL agent for Indian Rummy, trained on weak heuristics to beat strong search opponents at 7000x speed.
DeFAb is a large-scale, formally verifiable benchmark for defeasible abduction derived from 18 knowledge bases, demonstrating that frontier LLMs achieve 7.8-65% accuracy versus 100% for a rule-based solver with polynomial-time checks.
Machine learning discovers a tube-seeding strategy for IBP reduction of Feynman integrals that scales linearly with numerator power, demonstrated on rank-20 2-loop 5-point integrals.
A reward-free representation learning pipeline for offline PbRL achieves better preference efficiency than standard two-stage baselines by connecting RFRL concepts to preference data.
Dynamic isotropy, quantifying uniform center-of-mass acceleration capability, improves robot performance and enables omnidirectional locomotion, terrain traversal, and failure resilience in a spherical robot design.
AtomComposer uses online RL with multi-composition training to discover up to 10x more valid 3D isomers on unseen chemical formulas than single-composition baselines.
Agent-BRACE improves LLM agent performance on long-horizon partially observable tasks by 5.3-14.5% through a decoupled belief state of verbalized atomic claims with certainty labels that keeps context length constant.
SimWorld Studio deploys an evolving coding agent to create adaptive 3D environments that co-evolve with embodied learners, delivering 18-point success-rate gains over fixed environments in navigation benchmarks.
ReLibra uses pre-known token-to-expert routing from RL rollouts to perform inter-batch expert reordering and intra-batch replication, delivering up to 1.6x higher throughput than Megatron-LM and 1.2x over oracle-equipped EPLB while staying within 6-10% of an ideal balanced baseline.
Weak-to-strong generalization is nearly inevitable in linear logistic regression for most student-teacher pairs without any model capacity mismatch.
Observation and action delays are formally equivalent in cooperative Dec-POMDPs, yielding identical optimal solutions and enabling zero-shot transfer, though learning dynamics differ due to credit assignment and operational constraints.
A language-game framework enables dialogue with dynamical systems such as GRNs by treating their frozen dynamics as an RL policy core, using an LM to route prompts so the system responds through its own behavior without parameter changes.
RefereeBench shows that even the strongest video MLLMs reach only around 60% accuracy on multi-sport refereeing tasks and struggle with rule application and temporal grounding.
OP-GRPO is the first off-policy GRPO method for flow-matching models that reuses trajectories via replay buffer and importance sampling corrections, matching on-policy performance with 34.2% of the training steps.
User-turn generation reveals that LLMs' interaction awareness is largely decoupled from task accuracy, remaining near zero in deterministic settings even as accuracy scales to 96.8% on GSM8K.
Derives an exact unbiased policy gradient for RL post-training of diffusion LLMs via entropy-guided step selection and one-step denoising rewards, achieving state-of-the-art results on coding and logical reasoning benchmarks.
A certified gradient-based method for contact-rich manipulation that quantifies smoothing-induced errors via set-valued discrepancies and incorporates them into analytical reachable sets for robust affine feedback policies.
First in-orbit demonstration of a DRL-trained AI satellite attitude controller that performs robust inertial pointing after sim-to-real transfer.
Develops and tests the first effective safeguard for analytic gradient-based provably safe RL, showing safe training on three control tasks without performance loss.
Flow-GRPO is the first online RL method for flow matching models, raising GenEval accuracy from 63% to 95% and text-rendering accuracy from 59% to 92% with little reward hacking.
citing papers explorer
-
Regularized Reward-Punishment Reinforcement Learning
Introduces KCPR and its deep form klDMP that couples reward and punishment policies via learned priors, yielding improved safety and stability in grid-world and Gazebo navigation tasks over DQN, SQL and softDMP.
-
PPO-EAL: Exact Augmented Lagrangian Proximal Policy Optimization for Safe Robotic Control
PPO-EAL integrates exact augmented Lagrangian optimization into PPO for safe robotic control, with claimed theoretical guarantees and better empirical safety-performance tradeoffs on several robot benchmarks including sim-to-real gear assembly.
-
Position: RL Researchers Need to Distinguish Between Solving Simulators and Using Simulators as a Proxy
RL researchers need to separate 'solving simulators' from 'using simulators as proxies' to prevent misleading conclusions about algorithm performance.
-
Internalizing the Future: A Unified Agentic Training Paradigm for World Model Planning
A three-stage training pipeline internalizes world-model simulation and success estimation in LLM agents for improved planning on search and math tasks.
-
DanceOPD: On-Policy Generative Field Distillation
DanceOPD routes samples across capability velocity fields in flow-matching models and trains via on-policy student-induced states to compose T2I, local editing, and global editing without mutual interference.
-
Designing Reward Signals for Portable Query Generation: A Case Study in Industrial Semantic Job Search
Empirical study of RLAIF for portable query generation finds reward shaping controls performance more than optimizer choice and a rule-based reward floor yields +0.147 quality gain.
-
Adaptive Utility driven Resource Orchestration for Resilient AI (AURORA-AI)
AURORA-AI combines HJB feedback control, Lyapunov monitoring, and a fairness-aware utility into an adaptive policy that reallocates resources across heterogeneous models and outperforms static, round-robin, greedy, LinUCB, and PPO baselines in a stress simulation with bias shocks, drift, and black-s
-
Confidence-Aware Tool Orchestration for Robust Video Understanding
Robust-TO integrates per-frame reliability scores into tool orchestration and a confidence-cost GRPO reward to improve video reasoning robustness under corruption.
-
Revisiting Action Factorization for Complex Action Spaces
Comparative study of action factorization methods for hybrid action spaces across PPO/SAC/DQN finds branching dueling architectures effective and auto-regressive methods highest performing, with new VDN-PPO and PPO-MIX variants outperforming other PPO approaches.
-
Racing a Wheeled Quadruped: Active Load Transfer Mitigation via Model Predictive Control
Hierarchical MPC-RL framework for wheeled quadruped racing reduces mean lateral load transfer by up to 44% and improves lap times by 8.7% in track tests.
-
NavIsaacLab: Generating Realistic Crowd via Parallel Robot Learning for Benchmarking Human-aware Navigation
NavIsaacLab presents a new simulation platform combining photo-realistic rendering, parallel GPU execution, and data-driven pedestrian models to benchmark and train human-aware robot navigation policies.
-
Deep Reinforcement Learning-Enhanced Event-Triggered Data-Driven Predictive Control for a 3D Cable-Driven Soft Robotic Arm
RL-ET-DeePC trains a model-free RL policy to trigger DeePC optimizations adaptively, cutting optimization frequency by up to 66% in simulation and 34% on hardware while preserving tracking accuracy for a soft robotic arm.
-
From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Models
The survey formalizes MLLM perception as a unified vision-language capability and traces its evolution via a new five-stage taxonomy while outlining future challenges.
-
FlowTTS-GRPO: Online Reinforcement Learning with Multi-Objective Reward Optimization for Flow-Matching Based Text-to-Speech
FlowTTS-GRPO applies online RL with weighted multi-objective rewards to flow-matching TTS models via ODE-to-SDE conversion, reporting gains in speaker similarity and perceptual quality on CosyVoice 3.0 and F5-TTS.
-
Scalable Multi-Task Data Generation via Reinforcement Learning for Language-Conditioned Bimanual Dexterous Manipulation
An RL data generation pipeline with generalizable rewards and language annotations produces diverse synthetic datasets that improve multi-task policy generalization on three bimanual manipulation tasks.
-
Prefix-Guided On-Policy Distillation: Mining Golden Trajectories from Rollouts
PG-OPD uses early prefix overlap to selectively continue only high-compatibility rollouts in on-policy distillation, reporting up to 4.8 accuracy points gained and 2.46x less training time on math benchmarks.
-
Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learning
Survey mapping RL techniques onto LLM training and highlighting gaps in value-based, off-policy, and bootstrapping methods.
-
Motion-Aware Reinforcement Learning For Object Localization
MARLNet adds a motion prior to observations and smoothness penalty to rewards in a PPO bounding-box agent, producing small gains (+0.011 on VOC, +0.007 on VisDrone) at IoU≥0.5 while exposing a reward-interference failure mode.
-
FAST: A Framework for Aligned Sampling and Training in Parallel Reinforcement Learning for Autonomous Driving
FAST uses Dynamic Parallel Sampling Alignment via virtual continuation and Scaled Mask-Padding Optimization to remove straggler bottlenecks in parallel RL, delivering 1.78x wall-clock speedup while preserving unbiasedness.
-
Sakana Fugu Technical Report
Sakana Fugu trains LLM orchestrators using fine-tuning, evolutionary algorithms, and RL to build query-adaptive multi-agent scaffolds, claiming SOTA results on benchmarks including SWE-Bench Pro and GPQA-Diamond.
-
Pose-Agnostic Robotic Functional Grasping via Observation-Action Canonicalization
AnyMug trains a single closed-loop visuomotor policy in simulation using observation-action canonicalization and deploys it zero-shot on a real robot for functional mug-handle grasping across poses.
-
Marginal Advantage Accumulation for Memory-Driven Agent Self-Evolution
MAA formalizes alignability and comparability conditions and uses differential signals, EMA accumulation, and semantic identity merging to enable cross-batch operation-level evidence accumulation, outperforming batch-level baselines in 14 of 16 settings while matching online methods.
-
Direct Advantage Estimation for Scalable and Sample-efficient Deep Reinforcement Learning
Extends DAE theory to POMDPs with minimal changes and introduces discrete latent dynamics to cut computational cost, with ALE experiments showing scalability and retained sample efficiency.
-
VIMPO: Value-Implicit Policy Optimization for LLMs
VIMPO derives a policy-implied value function from optimality conditions for critic-free RL in LLMs and shows gains over GRPO on math benchmarks.
-
Signature Methods for Optimal Market Making
Signature linearization reduces optimal market making to pseudo-linear optimization over expected signatures of augmented paths, with Sig-REINFORCE algorithm learning bid/ask quotes and outperforming PPO on Poisson and Hawkes arrival models.
-
CTS-MoE: Implicit Terrain Adaptation via Mixture-of-Experts for Perceptive Locomotion
CTS-MoE combines a dense MoE actor with perception-based gating and a multi-critic architecture to enable adaptive perceptive locomotion on discontinuous terrain in a single-stage teacher-student training setup.
-
EfficientRollout: System-Aware Self-Speculative Decoding for RL Rollouts
EfficientRollout applies self-speculative decoding with quantized drafter induction and system-aware acceptance policies to cut RL rollout latency up to 19.6% while preserving final model quality.
-
Online Reward-Punishment Learning from Fixed-Channel Perceptual Event Streams without Environment Rewards
OHIRL separates next-packet prediction, residual dynamics, a fixed recovery-positive evaluator, and policy learning to achieve high sign and action accuracy in reward-free perceptual tasks where standard reward proxies fail.
-
Benchmarking Action Spaces in Reinforcement Learning for Vision-based Robotic Manipulation
Joint velocity action space outperforms pose increment, pose velocity, and joint position increment for smoothness and performance in sim-to-real vision-based manipulation.
-
When Web Agents Finish but Still Fail: Reproducible Triggers and Trace Diagnostics for Parallel Web Exploration
Parallel WebBench reveals GRPO training raises web agent completion to 96% but leaves a large correctness gap from context-bound loops, premature termination, and synthesis collapse.
-
Sparsity Curse: Understanding RLVR Model Parameter Space from Model Merging
RLVR induces sparse off-principal updates forming near-orthogonal shortcuts that degrade merging, addressed via Sensitivity-aware Resolving Merging using Fisher sensitivity, sparsification, and rescaling.
-
EnvRL: Learn from Environment Dynamics in Agentic Reinforcement Learning
EnvRL incorporates environment dynamics learning via state prediction and inverse dynamics auxiliary objectives into agentic RL, reporting higher success rates than RL-only baselines on ALFWorld and WebShop.
-
Embodiment Shapes Rolling Behavior in a Multimodal Infant Model
A reinforcement learning model of a multimodal virtual infant produces rolling behaviors that reproduce age-related improvements and coordination patterns observed in human infants, shaped by changing body morphology.
-
TerraTransfer: Learning End-to-End Driving Policies Without Expert Demonstrations
TerraTransfer decouples self-play policy pretraining from vision alignment via KL divergence and low-rank loss to produce end-to-end driving policies without expert demonstrations, matching prior methods on closed-loop photorealistic scenarios.
-
RISE: Relay Inference and Online Scheduling for Efficient Edge-Device Collaborative Diffusion Model Services
RISE introduces a training-free relay inference mechanism for diffusion models across edge and device plus a contextual bandit scheduler, reporting up to 2.1x speedup with preserved quality on two benchmarks.
-
GeoDisaster: Benchmarking Orchestrated Agents for Operational Disaster Geo-Intelligence
GeoDisaster provides a new benchmark for operational disaster geo-intelligence and proposes an RCEA-trained multi-agent framework with 18 geospatial tools that improves tool use and decision consistency over existing RS-VLMs.
-
TopoRetarget: Interaction-Preserving Retargeting for Dexterous Manipulation
TopoRetarget uses a sparse interaction graph and distance-weighted Laplacian deformation optimization with kinematic and penetration constraints to retarget human demonstrations to dexterous hands while preserving task-relevant contacts.
-
Human-like autonomy emerges from self-play and a pinch of human data
Self-play RL regularized with 30 minutes of human data produces driving policies that coordinate with humans, training in 15 hours on one GPU with 2500x less data than imitation learning.
-
Reward Modeling for Multi-Agent Orchestration
OrchRM uses intermediate artifacts from multi-agent runs to create training pairs for a reward model that guides orchestrator training and test-time scaling, reporting up to 10x token efficiency and 8% accuracy gains across reasoning domains.
-
Reinforcement Learning for Neural Model Editing
Neural model editing is cast as RL in MaskWorld and ShiftWorld environments, with learned policies reaching near-zero forget accuracy while retaining over 90% on retain sets for unlearning and improving bias metrics by over 5%.
-
GeoHAT: Geometry-Adaptive Hybrid Action Transformer for Mobile Manipulation
GeoHAT reports a 79.3% mean success rate on the ManiSkill-HAB mobile manipulation benchmark (23.7% above the strongest baseline) by using gated geometric token injection and a hybrid whole-body action decoder.
-
Keep Policy Gradient in Charge: Sibling-Guided Credit Distillation for Long-Horizon Tool-Use Agents
SGCD improves held-out scores on AppWorld and tau^3-airline by using LLM-summarized sibling contrasts to reshape GRPO advantages while keeping policy gradient in charge of the actor update.
-
KinematicRL: A Sim-to-Real Reinforcement Learning Framework For Social Navigation With Kinodynamic Feasibility
KinematicRL is a sim-to-real RL framework for social navigation using second-order control inputs, iLQR pretraining, and cluster-based 2D LiDAR tracking to produce kinodynamically feasible policies deployable on real robots with minimal modifications.
-
Space-sampled Value Decay: Forgetting Mechanisms for Non-stationary Deep Reinforcement Learning
Space-sampled Value Decay is proposed as a simple forgetting mechanism for DQN and SAC modifications that shows positive but limited effects on returns in non-stationary RL environments.
-
TacCoRL: Integrating Tactile Feedback into VLA via Simulation
TacCoRL integrates tactile feedback into VLA policies via real-aligned simulation co-training and RL, raising average success from 50% to 72.5% on four bimanual contact-rich tasks with direct real-robot transfer.
-
ReCal: Reward Calibration for RL-based LLM Routing
ReCal introduces hierarchical reward decomposition and distribution-aware optimization to address ambiguous credit assignment and optimization bias in RL-based LLM routing.
-
IAPO: Input Attribution-Aware Policy Optimization for Tool Use in Small Multimodal Agents
IAPO is an RL method that aligns model input attributions with a teacher to improve tool-calling in multimodal SLMs, reporting 3% average VQA accuracy gains on Qwen2.5-VL-3B across six tests.
-
LUCID: Learning Embodiment-Agnostic Intent Models from Unstructured Human Videos for Scalable Dexterous Robot Skill Acquisition
LUCID learns embodiment-agnostic intent models from unstructured human videos to train dexterous robot policies in simulation, enabling zero-shot transfer on real-world tasks like stirring and wiping.
-
PLUME: Probabilistic Latent Unified World Modeling and Parameter Estimation for Multi-Finger Manipulation
PLUME jointly models parameter beliefs and conditioned dynamics in a latent space for dexterous manipulation, enabling zero-shot sim-to-real transfer that outperforms offline RL and behavior cloning baselines on turning, lifting, and flicking tasks.
-
Bellman-Taylor Score Decoding for Markov Decision Processes with State-Dependent Feasible Action Sets
Bellman-Taylor score decoding framework for MDPs with implicit state-dependent action constraints, enabling standard DRL optimization with a decomposed optimality gap guarantee.