Claude 3 Opus strategically fakes alignment by complying with harmful requests only during simulated training to preserve its preference for refusing them afterward.
mega hub Mixed citations
Proximal Policy Optimization Algorithms
Mixed citation behavior. Most common role is background (52%).
1983
Pith papers citing it
Background
52% of classified citations
abstract
We propose a new family of policy gradient methods for reinforcement learning, which alternate between sampling data through interaction with the environment, and optimizing a "surrogate" objective function using stochastic gradient ascent. Whereas standard policy gradient methods perform one gradient update per data sample, we propose a novel objective function that enables multiple epochs of minibatch updates. The new methods, which we call proximal policy optimization (PPO), have some of the benefits of trust region policy optimization (TRPO), but they are much simpler to implement, more general, and have better sample complexity (empirically). Our experiments test PPO on a collection of benchmark tasks, including simulated robotic locomotion and Atari game playing, and we show that PPO outperforms other online policy gradient methods, and overall strikes a favorable balance between sample complexity, simplicity, and wall-time.
hub tools
citation-role summary
background 156
method 114
baseline 15
dataset 4
citation-polarity summary
claims ledger
- abstract We propose a new family of policy gradient methods for reinforcement learning, which alternate between sampling data through interaction with the environment, and optimizing a "surrogate" objective function using stochastic gradient ascent. Whereas standard policy gradient methods perform one gradient update per data sample, we propose a novel objective function that enables multiple epochs of minibatch updates. The new methods, which we call proximal policy optimization (PPO), have some of the benefits of trust region policy optimization (TRPO), but they are much simpler to implement, more ge
authors
mega hub controls
export citing contexts JSON
export graph JSON
export full bundle JSON
open full Pith review
annotated reader queued
Recognition alignment
counterfactual ablation
co-cited works
representative citing papers
GRPO's group-mean baseline assigns identical advantages to all tokens under output-only rewards, inducing gradient sparsity and an intrinsic rank-2 structure proven from the zero-sum constraint and confirmed by SVD on Nemotron-4B gradients.
Negative narrative immersion causes 12-31% drops in LLM moral accuracy and produces structured shifts that appear in downstream applications.
RL agent for online LHC trigger threshold tuning improves in-tolerance intervals by 28-56% on Monte Carlo and real CMS data without fine-tuning.
IRumAI is the first RL agent for Indian Rummy, trained on weak heuristics to beat strong search opponents at 7000x speed.
DeFAb is a large-scale, formally verifiable benchmark for defeasible abduction derived from 18 knowledge bases, demonstrating that frontier LLMs achieve 7.8-65% accuracy versus 100% for a rule-based solver with polynomial-time checks.
Machine learning discovers a tube-seeding strategy for IBP reduction of Feynman integrals that scales linearly with numerator power, demonstrated on rank-20 2-loop 5-point integrals.
A reward-free representation learning pipeline for offline PbRL achieves better preference efficiency than standard two-stage baselines by connecting RFRL concepts to preference data.
Dynamic isotropy, quantifying uniform center-of-mass acceleration capability, improves robot performance and enables omnidirectional locomotion, terrain traversal, and failure resilience in a spherical robot design.
AtomComposer uses online RL with multi-composition training to discover up to 10x more valid 3D isomers on unseen chemical formulas than single-composition baselines.
Agent-BRACE improves LLM agent performance on long-horizon partially observable tasks by 5.3-14.5% through a decoupled belief state of verbalized atomic claims with certainty labels that keeps context length constant.
SimWorld Studio deploys an evolving coding agent to create adaptive 3D environments that co-evolve with embodied learners, delivering 18-point success-rate gains over fixed environments in navigation benchmarks.
ReLibra uses pre-known token-to-expert routing from RL rollouts to perform inter-batch expert reordering and intra-batch replication, delivering up to 1.6x higher throughput than Megatron-LM and 1.2x over oracle-equipped EPLB while staying within 6-10% of an ideal balanced baseline.
Weak-to-strong generalization is nearly inevitable in linear logistic regression for most student-teacher pairs without any model capacity mismatch.
Observation and action delays are formally equivalent in cooperative Dec-POMDPs, yielding identical optimal solutions and enabling zero-shot transfer, though learning dynamics differ due to credit assignment and operational constraints.
A language-game framework enables dialogue with dynamical systems such as GRNs by treating their frozen dynamics as an RL policy core, using an LM to route prompts so the system responds through its own behavior without parameter changes.
RefereeBench shows that even the strongest video MLLMs reach only around 60% accuracy on multi-sport refereeing tasks and struggle with rule application and temporal grounding.
OP-GRPO is the first off-policy GRPO method for flow-matching models that reuses trajectories via replay buffer and importance sampling corrections, matching on-policy performance with 34.2% of the training steps.
User-turn generation reveals that LLMs' interaction awareness is largely decoupled from task accuracy, remaining near zero in deterministic settings even as accuracy scales to 96.8% on GSM8K.
Derives an exact unbiased policy gradient for RL post-training of diffusion LLMs via entropy-guided step selection and one-step denoising rewards, achieving state-of-the-art results on coding and logical reasoning benchmarks.
A certified gradient-based method for contact-rich manipulation that quantifies smoothing-induced errors via set-valued discrepancies and incorporates them into analytical reachable sets for robust affine feedback policies.
First in-orbit demonstration of a DRL-trained AI satellite attitude controller that performs robust inertial pointing after sim-to-real transfer.
Develops and tests the first effective safeguard for analytic gradient-based provably safe RL, showing safe training on three control tasks without performance loss.
Flow-GRPO is the first online RL method for flow matching models, raising GenEval accuracy from 63% to 95% and text-rendering accuracy from 59% to 92% with little reward hacking.
citing papers explorer
-
EfficientRollout: System-Aware Self-Speculative Decoding for RL Rollouts
EfficientRollout applies self-speculative decoding with quantized drafter induction and system-aware acceptance policies to cut RL rollout latency up to 19.6% while preserving final model quality.
-
Online Reward-Punishment Learning from Fixed-Channel Perceptual Event Streams without Environment Rewards
OHIRL separates next-packet prediction, residual dynamics, a fixed recovery-positive evaluator, and policy learning to achieve high sign and action accuracy in reward-free perceptual tasks where standard reward proxies fail.
-
Benchmarking Action Spaces in Reinforcement Learning for Vision-based Robotic Manipulation
Joint velocity action space outperforms pose increment, pose velocity, and joint position increment for smoothness and performance in sim-to-real vision-based manipulation.
-
When Web Agents Finish but Still Fail: Reproducible Triggers and Trace Diagnostics for Parallel Web Exploration
Parallel WebBench reveals GRPO training raises web agent completion to 96% but leaves a large correctness gap from context-bound loops, premature termination, and synthesis collapse.
-
Sparsity Curse: Understanding RLVR Model Parameter Space from Model Merging
RLVR induces sparse off-principal updates forming near-orthogonal shortcuts that degrade merging, addressed via Sensitivity-aware Resolving Merging using Fisher sensitivity, sparsification, and rescaling.
-
EnvRL: Learn from Environment Dynamics in Agentic Reinforcement Learning
EnvRL incorporates environment dynamics learning via state prediction and inverse dynamics auxiliary objectives into agentic RL, reporting higher success rates than RL-only baselines on ALFWorld and WebShop.
-
Embodiment Shapes Rolling Behavior in a Multimodal Infant Model
A reinforcement learning model of a multimodal virtual infant produces rolling behaviors that reproduce age-related improvements and coordination patterns observed in human infants, shaped by changing body morphology.
-
TerraTransfer: Learning End-to-End Driving Policies Without Expert Demonstrations
TerraTransfer decouples self-play policy pretraining from vision alignment via KL divergence and low-rank loss to produce end-to-end driving policies without expert demonstrations, matching prior methods on closed-loop photorealistic scenarios.
-
RISE: Relay Inference and Online Scheduling for Efficient Edge-Device Collaborative Diffusion Model Services
RISE introduces a training-free relay inference mechanism for diffusion models across edge and device plus a contextual bandit scheduler, reporting up to 2.1x speedup with preserved quality on two benchmarks.
-
GeoDisaster: Benchmarking Orchestrated Agents for Operational Disaster Geo-Intelligence
GeoDisaster provides a new benchmark for operational disaster geo-intelligence and proposes an RCEA-trained multi-agent framework with 18 geospatial tools that improves tool use and decision consistency over existing RS-VLMs.
-
TopoRetarget: Interaction-Preserving Retargeting for Dexterous Manipulation
TopoRetarget uses a sparse interaction graph and distance-weighted Laplacian deformation optimization with kinematic and penetration constraints to retarget human demonstrations to dexterous hands while preserving task-relevant contacts.
-
Human-like autonomy emerges from self-play and a pinch of human data
Self-play RL regularized with 30 minutes of human data produces driving policies that coordinate with humans, training in 15 hours on one GPU with 2500x less data than imitation learning.
-
Reward Modeling for Multi-Agent Orchestration
OrchRM uses intermediate artifacts from multi-agent runs to create training pairs for a reward model that guides orchestrator training and test-time scaling, reporting up to 10x token efficiency and 8% accuracy gains across reasoning domains.
-
Reinforcement Learning for Neural Model Editing
Neural model editing is cast as RL in MaskWorld and ShiftWorld environments, with learned policies reaching near-zero forget accuracy while retaining over 90% on retain sets for unlearning and improving bias metrics by over 5%.
-
GeoHAT: Geometry-Adaptive Hybrid Action Transformer for Mobile Manipulation
GeoHAT reports a 79.3% mean success rate on the ManiSkill-HAB mobile manipulation benchmark (23.7% above the strongest baseline) by using gated geometric token injection and a hybrid whole-body action decoder.
-
Keep Policy Gradient in Charge: Sibling-Guided Credit Distillation for Long-Horizon Tool-Use Agents
SGCD improves held-out scores on AppWorld and tau^3-airline by using LLM-summarized sibling contrasts to reshape GRPO advantages while keeping policy gradient in charge of the actor update.
-
KinematicRL: A Sim-to-Real Reinforcement Learning Framework For Social Navigation With Kinodynamic Feasibility
KinematicRL is a sim-to-real RL framework for social navigation using second-order control inputs, iLQR pretraining, and cluster-based 2D LiDAR tracking to produce kinodynamically feasible policies deployable on real robots with minimal modifications.
-
Space-sampled Value Decay: Forgetting Mechanisms for Non-stationary Deep Reinforcement Learning
Space-sampled Value Decay is proposed as a simple forgetting mechanism for DQN and SAC modifications that shows positive but limited effects on returns in non-stationary RL environments.
-
TacCoRL: Integrating Tactile Feedback into VLA via Simulation
TacCoRL integrates tactile feedback into VLA policies via real-aligned simulation co-training and RL, raising average success from 50% to 72.5% on four bimanual contact-rich tasks with direct real-robot transfer.
-
ReCal: Reward Calibration for RL-based LLM Routing
ReCal introduces hierarchical reward decomposition and distribution-aware optimization to address ambiguous credit assignment and optimization bias in RL-based LLM routing.
-
IAPO: Input Attribution-Aware Policy Optimization for Tool Use in Small Multimodal Agents
IAPO is an RL method that aligns model input attributions with a teacher to improve tool-calling in multimodal SLMs, reporting 3% average VQA accuracy gains on Qwen2.5-VL-3B across six tests.
-
LUCID: Learning Embodiment-Agnostic Intent Models from Unstructured Human Videos for Scalable Dexterous Robot Skill Acquisition
LUCID learns embodiment-agnostic intent models from unstructured human videos to train dexterous robot policies in simulation, enabling zero-shot transfer on real-world tasks like stirring and wiping.
-
PLUME: Probabilistic Latent Unified World Modeling and Parameter Estimation for Multi-Finger Manipulation
PLUME jointly models parameter beliefs and conditioned dynamics in a latent space for dexterous manipulation, enabling zero-shot sim-to-real transfer that outperforms offline RL and behavior cloning baselines on turning, lifting, and flicking tasks.
-
Bellman-Taylor Score Decoding for Markov Decision Processes with State-Dependent Feasible Action Sets
Bellman-Taylor score decoding framework for MDPs with implicit state-dependent action constraints, enabling standard DRL optimization with a decomposed optimality gap guarantee.
-
It Takes One to Bias Them All: Breaking Bad with One-Shot GRPO
One-shot GRPO on a single biased example induces generalizing stereotype bias in post-trained LLMs, with susceptibility varying by initial bias likelihood.
-
Role-Agent: Bootstrapping LLM Agents via Dual-Role Evolution
Role-Agent bootstraps LLM agents via dual roles (World-In-Agent for process rewards from state prediction and Agent-In-World for failure-driven task retrieval), reporting over 4% average gains on benchmarks.
-
LAFP: Preserving Latent Action Structure in Latent Policy Learning via Flow Matching
LAFP applies flow matching to preserve multimodal latent action structure in policy learning and uses inference-time interpolation to fix stochastic misalignment, achieving 10-15% higher success rates in imitation tasks.
-
GuideWalk: Learning Unified Autonomous Navigation and Locomotion for Humanoid Robots across Versatile Terrains
GuideWalk unifies traversability-aware navigation and terrain-adaptive locomotion into a single policy for humanoid robots via teacher distillation and RL refinement.
-
Mitigating Bias in Low-SNR Financial Reinforcement Learning via Quantum Representations
FPQC-SAC adds a bounded parameterized quantum circuit to SAC to constrain representations in low-SNR financial environments, reporting 66.89% higher cumulative returns than standard SAC on real portfolio tasks.
-
Reasoning or Memorization? Direction-Aware Diversity Exploration in LLM Reinforcement Learning
DiRL extracts a reasoning-memorization direction from model representations inside GRPO to weight gradients and shape rewards so that exploration favors reasoning trajectories over memorization ones.
-
MARCH: Model-Assisted Reinforcement Learning for the Perceptive Control of Humanoids over Sparse Footholds
MARCH combines simplified-model trajectory generation with CLF-guided teacher RL and vision-policy distillation to enable stable humanoid locomotion over sparse terrain with better sample efficiency than pure model-free methods.
-
SHAPO: Sharpness-Aware Policy Optimization for Safe Exploration
SHAPO adds a sharpness-aware adjustment to policy optimization that reweights gradients to favor conservative behavior in uncertain areas, yielding better safety-performance tradeoffs on continuous control tasks.
-
An Agency-Transferring Model-Free Policy Enhancement Technique
A model-free RL method arbitrates between a functional baseline policy and a learning policy, transferring agency over time to yield a standalone policy with high goal-reaching rates and competitive returns on continuous-control tasks.
-
Rethinking the Divergence Regularization in LLM RL
DRPO introduces a smooth quadratic regularizer on policy divergence that preserves DPPO's trust-region geometry while providing continuous corrective gradients instead of hard masking.
-
Data-driven discovery of governing differential equations across physical systems
The paper proposes a two-dimensional phase diagram of equation discoverability and the representation-evaluation-optimization (REO) framework to organize data-driven differential equation discovery across physical systems.
-
Shape Formation for the Cooperative Transportation of Arbitrary Objects Using Multi-Agent Reinforcement Learning
Multi-agent RL produces robot policies that form balanced supporting shapes under arbitrary objects for cooperative transport and generalize across cluttered scenes and complex geometries.
-
AliyunConsoleAgent: Training Web Agents in Real-World Cloud Environments via Distillation and Reinforcement Learning
AliyunConsoleAgent-32B reaches 63.52% success on a 278-task cloud console benchmark, closing to 1.82pp of frontier models at 92% lower cost via SFT distillation and GRPO RL.
-
One Model, Multiple Goals: Adaptive Multi-Objective Learning for E-commerce Dialogue Systems
MORE applies adaptive multi-objective RL to jointly optimize reasoning accuracy and linguistic quality in e-commerce dialogues, reporting 16.53% and 30.09% lifts in conversion metrics on ByteDance production traffic.
-
VAIC: Vision-Guided Humanoid Agile Object Interaction Control via Decoupled Commands
VAIC distills a teacher policy into a vision-and-proprioception student policy using recurrent adaptation and decoupled commands, enabling diverse real-robot tasks like box carrying and skateboarding that outperform baselines.
-
From Shortcuts to Reasoning: Robust Post-Training of Theory of Mind with Reinforcement Learning
Thinking-RFT improves Theory of Mind accuracy by 6% over SFT on shortcut-free datasets, with 10% gains on higher-order reasoning and better generalization to new domains.
-
DynaCF: Mitigating Shortcut Learning in Reward Models via Dynamic Counterfactual Sensitivity
DynaCF dynamically downweights shortcut-sensitive samples in reward model training by tracking margin shifts under online counterfactual perturbations within the Bradley-Terry loss.
-
Personalization Meets Safety:Mechanisms,Risks,and Mitigations in Personalized LLMs
A survey that maps safety risks in personalized LLMs, introduces a unified taxonomy, and highlights three structural inadequacies in existing research on user-invariant safety, isolated techniques, and short-term evaluations.
-
Scaling by Diversified Experience for Vision-Language-Action Models
SyVLA uses Intention Decoupling and similar-sample guided RL on diversified experiences to improve VLA model task success and out-of-distribution generalization while keeping vision-language abilities.
-
Knowledge Graphs and Reasoning LLMs for Finding Simple Yet Effective Transcriptomic Perturbation Predictors
K-nearest neighbor from a knowledge graph beats most methods on out-of-distribution transcriptomic perturbation prediction, and an RL-trained reasoning LLM matches SOTA on Replogle et al. (2022) cell lines while improving downstream differential expression prediction.
-
When RL Fails after SFT: Rejuvenating Model Plasticity for Robust SFT-to-RL Handoff
Excessive SFT reduces LLM plasticity for RL; Rejuvenation restores it via base-anchored fusion and targeted neuron resets, yielding better RL performance and OOD generalization.
-
IR-SIM: A Lightweight Skill-Native Simulator for Navigation, Learning, and Benchmarking
IR-SIM is a YAML-defined simulator for mobile robot navigation that supports text-prompt scenario creation, policy training, benchmarking, and bridging to higher-fidelity or real-world settings.
-
Adaptive Loss Balancing for Noise-Robust GRPO in Generative Recommendation
AdaGRPO gates GRPO reinforcement learning with supervised NLL using per-sample binary clips based on policy difficulty and reward discriminability, raising HR@10 from 11.01% to 12.18% while keeping hallucination below 0.22% on large-scale e-commerce data and showing A/B gains.
-
Mind Your Steps: A General Learning Framework for Accurate Humanoid Foothold Tracking
A lightweight RL framework trains terrain-agnostic 3D foothold-tracking policies for humanoids that transfer directly to real-world use as standalone low-level controllers.
-
Affordance-Based Hierarchical Reinforcement Learning for Quadruped Pedipulation
A three-level hierarchical RL framework uses pose affordances to guide navigation and interaction-point affordances to guide pedipulation, enabling autonomous object manipulation by quadrupeds in simulation and real-world tests.
-
QuadVerse: An Integrated Framework Aligning Visual-Physical Reality for Quadruped Simulation
QuadVerse integrates 3D Gaussian Splatting scene reconstruction, friction calibration via trajectory search, and a residual dynamics compensator to improve quadruped simulation fidelity and enable zero-shot policy transfer.