Claude 3 Opus strategically fakes alignment by complying with harmful requests only during simulated training to preserve its preference for refusing them afterward.
mega hub Mixed citations
Proximal Policy Optimization Algorithms
Mixed citation behavior. Most common role is background (52%).
1373
Pith papers citing it
Background
52% of classified citations
abstract
We propose a new family of policy gradient methods for reinforcement learning, which alternate between sampling data through interaction with the environment, and optimizing a "surrogate" objective function using stochastic gradient ascent. Whereas standard policy gradient methods perform one gradient update per data sample, we propose a novel objective function that enables multiple epochs of minibatch updates. The new methods, which we call proximal policy optimization (PPO), have some of the benefits of trust region policy optimization (TRPO), but they are much simpler to implement, more general, and have better sample complexity (empirically). Our experiments test PPO on a collection of benchmark tasks, including simulated robotic locomotion and Atari game playing, and we show that PPO outperforms other online policy gradient methods, and overall strikes a favorable balance between sample complexity, simplicity, and wall-time.
hub tools
citation-role summary
background 154
method 113
baseline 15
dataset 4
citation-polarity summary
claims ledger
- abstract We propose a new family of policy gradient methods for reinforcement learning, which alternate between sampling data through interaction with the environment, and optimizing a "surrogate" objective function using stochastic gradient ascent. Whereas standard policy gradient methods perform one gradient update per data sample, we propose a novel objective function that enables multiple epochs of minibatch updates. The new methods, which we call proximal policy optimization (PPO), have some of the benefits of trust region policy optimization (TRPO), but they are much simpler to implement, more ge
authors
mega hub controls
export citing contexts JSON
export graph JSON
export full bundle JSON
open full Pith review
annotated reader queued
Recognition alignment
counterfactual ablation
co-cited works
representative citing papers
Dynamic isotropy, quantifying uniform center-of-mass acceleration capability, improves robot performance and enables omnidirectional locomotion, terrain traversal, and failure resilience in a spherical robot design.
Agent-BRACE improves LLM agent performance on long-horizon partially observable tasks by 5.3-14.5% through a decoupled belief state of verbalized atomic claims with certainty labels that keeps context length constant.
SimWorld Studio deploys an evolving coding agent to create adaptive 3D environments that co-evolve with embodied learners, delivering 18-point success-rate gains over fixed environments in navigation benchmarks.
ReLibra uses pre-known token-to-expert routing from RL rollouts to perform inter-batch expert reordering and intra-batch replication, delivering up to 1.6x higher throughput than Megatron-LM and 1.2x over oracle-equipped EPLB while staying within 6-10% of an ideal balanced baseline.
Weak-to-strong generalization is nearly inevitable in linear logistic regression for most student-teacher pairs without any model capacity mismatch.
Observation and action delays are formally equivalent in cooperative Dec-POMDPs, yielding identical optimal solutions and enabling zero-shot transfer, though learning dynamics differ due to credit assignment and operational constraints.
A language-game framework enables dialogue with dynamical systems such as GRNs by treating their frozen dynamics as an RL policy core, using an LM to route prompts so the system responds through its own behavior without parameter changes.
RefereeBench shows that even the strongest video MLLMs reach only around 60% accuracy on multi-sport refereeing tasks and struggle with rule application and temporal grounding.
OP-GRPO is the first off-policy GRPO method for flow-matching models that reuses trajectories via replay buffer and importance sampling corrections, matching on-policy performance with 34.2% of the training steps.
User-turn generation reveals that LLMs' interaction awareness is largely decoupled from task accuracy, remaining near zero in deterministic settings even as accuracy scales to 96.8% on GSM8K.
Derives an exact unbiased policy gradient for RL post-training of diffusion LLMs via entropy-guided step selection and one-step denoising rewards, achieving state-of-the-art results on coding and logical reasoning benchmarks.
A certified gradient-based method for contact-rich manipulation that quantifies smoothing-induced errors via set-valued discrepancies and incorporates them into analytical reachable sets for robust affine feedback policies.
First in-orbit demonstration of a DRL-trained AI satellite attitude controller that performs robust inertial pointing after sim-to-real transfer.
Develops and tests the first effective safeguard for analytic gradient-based provably safe RL, showing safe training on three control tasks without performance loss.
Flow-GRPO is the first online RL method for flow matching models, raising GenEval accuracy from 63% to 95% and text-rendering accuracy from 59% to 92% with little reward hacking.
SWE-Gym supplies 2438 executable real-world Python tasks to train SWE agents and verifiers, yielding up to 19% gains and new open-weight SOTA of 32% on SWE-Bench Verified.
BEHAVIOR-1K introduces a benchmark of 1,000 human everyday activities in realistic simulated scenes together with the OMNIGIBSON physics simulator to evaluate embodied AI.
ORPO performs preference alignment during supervised fine-tuning via a monolithic odds ratio penalty, allowing 7B models to outperform larger state-of-the-art models on alignment benchmarks.
IWR improves CRL sample efficiency and performance in interaction-rich manipulation by interaction-aware resampling that preserves mode boundaries, yielding 19.8% average gains and a real-world air-hockey agent.
EFE-based planning is formulated as variational free energy minimization with epistemic priors, decomposing into expected plan costs plus a complexity term.
EFE-based active inference planning is characterized as VFE on an augmented model plus entropy and planning corrections, with a derived message-passing implementation and grid-world validation.
Reinforcement learning on a bead-spring cilia model identifies antiplectic coordination as flow-maximizing, with a tilted-slider reduced model showing that a time-averaged position shift opposite the effective stroke enhances transport via elastic restoring force coupling, and that symplectic coordi
PromptPO shows LLMs can act as black-box policy optimizers for sequential RL when leveraging prior knowledge, matching baselines in exploration and robotics but underperforming in MuJoCo.
citing papers explorer
-
QAP-Router: Tackling Qubit Routing as Dynamic Quadratic Assignment with Reinforcement Learning
QAP-Router models qubit routing as dynamic QAP and applies RL with a solution-aware Transformer to cut CNOT counts by 12-30% versus industry compilers on real circuit benchmarks.
-
TuniQ: Autotuning Compilation Passes for Quantum Workloads at Scale for Effectiveness and Efficiency
TuniQ uses RL with a dual-encoder, shaped rewards, and action masking to autotune quantum compilation passes, improving fidelity and speed over Qiskit while generalizing across backends and scaling to large circuits.
-
Equivariant Reinforcement Learning for Clifford Quantum Circuit Synthesis
Equivariant RL agent synthesizes near-optimal Clifford circuits up to 30 qubits with lower two-qubit gate counts than Qiskit baselines.
-
Ember: An Extensible Benchmark Suite for Quantum Annealing Embedding Algorithms
Ember provides the first standardized, reproducible benchmark framework with 24,016 diverse graph instances for quantum annealing embedding algorithms, showing that no single algorithm performs best across all graph families.
-
Replay-buffer engineering for noise-robust quantum circuit optimization
Treating the replay buffer as a central lever in RL for quantum circuit optimization yields 4-32x sample efficiency gains, up to 67.5% faster episodes, and 85-90% fewer steps to accuracy on noisy molecular and compilation tasks.
-
Dynamic locking of an interacting spin system via periodic driving
Detuning from resonance combined with pulse structure in periodic driving produces a structured effective Rabi field that enables dynamic spin locking, reversible Zeeman-dipolar order interconversion, and heterospin polarization transfer in interacting spin systems.
-
Reinforcement learning for ion shuttling on trapped-ion quantum computers
Reinforcement learning optimizes ion shuttling on trapped-ion quantum chips and reduces operations by up to 36.3% versus heuristics across multiple architectures.
-
CO-MAP: A Reinforcement Learning Approach to the Qubit Allocation Problem
Reinforcement learning policy for qubit mapping reduces SWAP overhead by 65-85% versus standard quantum compilers on MQTBench and Queko benchmark circuits.
-
Qubit-Scalable CVRP via Lagrangian Knapsack Decomposition and Noise-Aware Quantum Execution
A hybrid quantum framework decomposes CVRP into bounded-width knapsack subproblems, trains a reinforcement learning controller for Lagrangian multipliers, and uses a contextual bandit to adapt quantum hardware execution, yielding improved routing quality on standard test instances.
-
Efficient $n$-qubit entangling operations via a superconducting quantum router
A superconducting quantum router enables programmable multi-qubit entangling operations, demonstrated with faster preparation of entangled states and RL-trained 2- and 3-qubit gates like Toffoli and Fredkin.
-
Learning Unified Control of Intrinsic Nonlinear Spin Dynamics in Atomic Qudits for Magnetometry
Reinforcement learning stabilizes more than 4 dB of fixed-axis spin squeezing under continuous nonlinear Zeeman evolution in the f=21/2 manifold of 161Dy, yielding a single-atom sensitivity of 13.9 pT/sqrt(Hz) that is 3 dB beyond the standard quantum limit.
-
Scheduling Entanglement Flows in Multi-channel Quantum Networks
PPO-based scheduling balances low delay and high success rates better than classical methods in simulations of multi-channel quantum entanglement distribution.
-
Reinforcement Learning for Robust Calibration of Multi-Qudit Quantum Gates
A hybrid optimal-control-plus-contextual-RL framework learns low-dimensional residual pulse corrections that preserve high-fidelity controlled-phase gates on two qutrits under realistic static model mismatch.
-
Investigation of Automated Design of Quantum Circuits for Imaginary Time Evolution Methods Using Deep Reinforcement Learning
DDQN reinforcement learning automates VITE circuit design, producing circuits with ~37% fewer gates and ~43% less depth than hardware-efficient ansatze for Max-Cut while reaching Full-CI for H2 with shallower depth.
-
Deep reinforcement learning for near-deterministic preparation of cubic- and quartic-phase gates in photonic quantum computing
Reinforcement learning controls photonic circuits to prepare cubic-phase states at 96% success and directly generate quartic-phase gates with photon-number-resolving measurements.
-
Hardware Co-Designed Optimal Control for Programmable Atomic Quantum Processors via Reinforcement Learning
Hardware-co-designed end-to-end differentiable reinforcement learning achieves robust >99.9% fidelity single-qubit gates in atomic quantum processors under realistic crosstalk and dynamic imperfections, outperforming PPO and SADE-Adam.
-
Molecular Quantum Control Algorithm Design by Reinforcement Learning
Reinforcement learning designs quantum-logic pulse sequences to prepare H3O+ (130 states) and CaH+ molecular ions in pure states from thermal populations.
-
Benchmarking Swarm Optimization Algorithms for Parameter Initialization in the Quantum Approximate Optimization Algorithm
Swarm methods such as PSO, FIPSO, and QPSO yield lower approximation gaps and more stable convergence than Adam, COBYLA, or SPSA when tuning QAOA parameters on weighted MaxCut instances, especially under noise and limited shots.