pith. sign in

arxiv: 1707.06347 · v2 · submitted 2017-07-20 · 💻 cs.LG

Proximal Policy Optimization Algorithms

John Schulman , Filip Wolski , Prafulla Dhariwal , Alec Radford , Oleg Klimov This is my paper

Pith reviewed 2026-05-08 23:02 UTC · model claude-opus-4-7

classification 💻 cs.LG
keywords policy gradientreinforcement learningtrust regionclipped surrogate objectiveproximal policy optimizationactor-criticcontinuous controlAtari
0
0 comments X

The pith

A simple clip on the policy probability ratio reproduces the practical benefits of trust region policy optimization with first-order SGD.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper asks whether the practical advantages of trust region policy optimization — stable, large policy updates that do not destroy the policy — can be captured by a first-order method with no constrained optimization machinery. The answer it proposes is yes, via a single change to the loss: take the policy-gradient surrogate, written as a probability ratio times an advantage, and replace it with the minimum of the unclipped term and the same term with the ratio clipped to [1−ε, 1+ε]. Optimizing this objective with several epochs of minibatch SGD on each batch of trajectories is shown to outperform TRPO, A2C, vanilla policy gradient, and the cross-entropy method on a suite of MuJoCo locomotion tasks, and to be roughly comparable to ACER on Atari while being dramatically simpler to implement and compatible with shared policy/value networks. A second variant that adapts a KL-divergence penalty coefficient to hit a target KL is offered as a baseline and works almost as well. The contribution is mostly empirical and engineering-driven: a simpler, more general recipe that hits a favorable point on the sample-complexity / wall-time / implementation-effort frontier.

Core claim

The paper introduces a policy gradient objective in which the probability ratio between the new and old policy, multiplied by an advantage estimate, is clipped to a small interval around 1 and then combined with the unclipped term by taking a minimum. This single change converts a standard policy gradient loss into one that can safely be optimized for many epochs of minibatch SGD on the same batch of trajectories, because the clip removes the incentive to push the ratio far from 1 when doing so would help the surrogate. The authors argue this captures the practical benefit of trust region policy optimization — keeping each update inside an implicit trust region — without needing constrained

What carries the argument

The clipped surrogate L^CLIP(θ) = Ê_t[min(r_t(θ) Â_t, clip(r_t(θ), 1−ε, 1+ε) Â_t)], where r_t(θ) is the new-to-old policy probability ratio at sampled action a_t and Â_t is an advantage estimate (here from truncated generalized advantage estimation). The min-of-clipped-and-unclipped construction is the load-bearing object: it equals the unconstrained surrogate to first order around θ_old, but flattens out once the ratio leaves [1−ε, 1+ε] in the direction that would inflate the objective, while preserving the gradient when the ratio moves in the direction that worsens it. This asymmetry makes the loss a per-sample pessimistic bound on the conservative-policy-iteration surrogate, which the aut

If this is right

  • Policy gradient training can take many epochs of minibatch SGD per collected batch without the destructive updates that motivated TRPO, raising sample efficiency at constant wall-clock cost.
  • The clipped objective is fully compatible with shared policy/value architectures, dropout, and recurrent networks, removing one of TRPO's main practical limitations.
  • On Atari, a much simpler algorithm than ACER reaches comparable final scores and superior early-training scores, suggesting that experience replay and off-policy correction are not required for competitive on-policy performance at this scale.
  • An adaptive KL-penalty variant performs nearly as well, indicating the operative ingredient is bounding per-update policy change rather than the specific clip shape — leaving room for other proximal regularizers to play the same role.
  • The recipe — collect a batch with N parallel actors, run K epochs of clipped-surrogate SGD, repeat — gives a default RL training loop with very few moving parts, plausibly explaining why it would become a workhorse baseline.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The 'pessimistic lower bound' framing is per-sample and per-timestep rather than a true expectation-level bound on policy improvement, so the trust-region intuition is heuristic; whether the clip actually constrains average KL across an update appears to be an empirical observation, not a guarantee.
  • Because clipping zeros the gradient outside [1−ε, 1+ε], samples whose ratio drifts past the clip stop contributing for the rest of the epoch loop, so ε and the number of epochs K are coupled knobs that together act as an implicit early-stopping rule on each batch.
  • The hyperparameter search in Section 6.1 is performed on the same MuJoCo suite later used to report scores, so the headline ranking against TRPO and A2C should be read as in-distribution performance rather than as evidence of out-of-the-box robustness.
  • Three random seeds is thin for the ranking claims on 49 Atari games in Table 2; the win counts likely have substantial variance and a more thorough seed sweep could redistribute several games between PPO and ACER.

Load-bearing premise

That clipping the policy ratio acts as a reliable substitute for an actual trust region — the paper's lower-bound story is informal and only holds per-sample, so the method's stability across tasks rests on empirical observation from a few benchmarks and seeds rather than on a monotonic-improvement guarantee.

What would settle it

Re-run the MuJoCo and Atari benchmarks with more seeds and with hyperparameters frozen before any tuning on these same environments; if PPO with ε≈0.2 and K=10 epochs no longer outperforms TRPO and A2C on average, or if the clipped objective ceases to dominate the adaptive-KL and no-clip variants in Table 1, the central claim that clipping is the operative ingredient does not survive.

read the original abstract

We propose a new family of policy gradient methods for reinforcement learning, which alternate between sampling data through interaction with the environment, and optimizing a "surrogate" objective function using stochastic gradient ascent. Whereas standard policy gradient methods perform one gradient update per data sample, we propose a novel objective function that enables multiple epochs of minibatch updates. The new methods, which we call proximal policy optimization (PPO), have some of the benefits of trust region policy optimization (TRPO), but they are much simpler to implement, more general, and have better sample complexity (empirically). Our experiments test PPO on a collection of benchmark tasks, including simulated robotic locomotion and Atari game playing, and we show that PPO outperforms other online policy gradient methods, and overall strikes a favorable balance between sample complexity, simplicity, and wall-time.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

4 major / 8 minor

Summary. The paper introduces Proximal Policy Optimization (PPO), a family of first-order policy-gradient methods that perform multiple epochs of minibatch SGD on a surrogate objective per batch of on-policy data. Two surrogates are proposed: (i) a clipped probability-ratio objective L^CLIP (Eq. 7), which takes the minimum of the unclipped importance-weighted advantage and a version where the ratio is clipped to [1−ε, 1+ε]; and (ii) an adaptive KL-penalty variant L^KLPEN (Eq. 8) that adjusts β to track a target KL. The authors argue L^CLIP forms a pessimistic lower bound on the conservative-policy-iteration objective L^CPI, allowing safe multi-epoch optimization without TRPO's conjugate-gradient/line-search machinery. Empirically, on 7 MuJoCo tasks (1M steps, 3 seeds, hyperparameters from a sweep), PPO with ε=0.2 outperforms TRPO, A2C, A2C+TR, vanilla PG, and CEM (Fig. 3, Table 1); on 49 Atari games (40M frames), PPO beats A2C in average-reward-over-training and is competitive with ACER (Table 2, Appendix B). Roboschool 3D humanoid tasks are showcased as scaling demonstrations.

Significance. If the empirical claims hold, the contribution is a methodologically simple, broadly applicable on-policy algorithm that captures most of TRPO's reliability with a few lines of code change to vanilla policy gradients, and is compatible with parameter-sharing and recurrent architectures (where TRPO's Fisher-vector machinery is awkward). The clean ablation in Table 1 across clipping, fixed-KL, and adaptive-KL settings is useful and supports the design choice. The work is well-positioned to become a strong default baseline for continuous-control and discrete-action benchmarks. The Atari sweep over 49 games and the head-to-head with ACER (Table 2, Fig. 6) is a substantial empirical scope. The hyperparameter tables in Appendix A make the work straightforwardly reproducible. Limitations on the theoretical side (no monotonic-improvement theorem analogous to TRPO) are acknowledged in spirit but should be sharpened (see major comments).

major comments (4)
  1. [§3 (Eq. 7) — 'pessimistic lower bound' claim] The paper claims L^CLIP is a 'lower bound (i.e., pessimistic bound) on the unclipped objective.' This is true only sample-wise: for each t, min(r_t Â_t, clip(r_t,1±ε)Â_t) ≤ r_t Â_t. It does not entail a lower bound on the true policy improvement η(π) − η(π_old) analogous to TRPO/Kakade-Langford, nor does it bound E_t[KL]. Once r_t exits [1−ε,1+ε] on the favorable side, the clipped branch is selected and its gradient w.r.t. θ on that sample is exactly zero — there is no restoring force, only a removed incentive. After K epochs of minibatch SGD on the same batch (the very feature being sold in §5), individual samples can drift arbitrarily far outside the band while the still-unclipped samples continue to drive updates. Please either (a) state precisely what the lower-bound property does and does not imply, or (b) provide a formal statement (even a per-state one) that connects L^CLIP optimi
  2. [Algorithm 1 / §5 — what is actually being evaluated] Algorithm 1 as written contains no KL-based early stopping, no value-loss clipping, no advantage normalization, no learning-rate annealing for MuJoCo, and no orthogonal init. Several of these are commonly used in implementations the community calls 'PPO' and are known to affect stability. The central methodological claim is that L^CLIP is what gives PPO its TRPO-like reliability. To support that, please clarify which auxiliary techniques (if any) are present in the implementation that produced Fig. 3, Table 1, and Table 2, and ideally include an ablation showing the contribution of L^CLIP itself versus those auxiliaries. Without this, it is hard to attribute the empirical wins to the clipped objective per se.
  3. [§6.1, Table 1 and §6.2, Fig. 3 — hyperparameter selection vs. evaluation] ε, β, and d_targ are searched on the same 7 MuJoCo environments that are then used to report Table 1, and the chosen ε=0.2 is reused in Fig. 3 against TRPO, A2C, etc. on those same environments. This raises a generalization concern for the headline ranking. Please report (i) seed-level variability (only 3 seeds per environment for Fig. 3 is thin for ranking claims), e.g., confidence intervals or per-seed traces, and (ii) at least one held-out environment or a leave-one-environment-out check that the chosen ε transfers. The Atari results (Table 2) partly address this with α-annealed ε=0.1, but the MuJoCo headline claim does not.
  4. [§6.4, Table 2 — divergent metric outcomes] Under metric (1) (avg reward over all training) PPO wins 30 to ACER's 18; under metric (2) (last 100 episodes) ACER wins 28 to PPO's 19. The paper presents both numbers but does not discuss the implication: PPO learns faster but ACER reaches better final performance on a majority of games. The current narrative ('competitive with ACER though much simpler') understates this. Please discuss and, if possible, report the metric (2) gap with effect sizes; with only 3 seeds per game (Fig. 6), I would also like to see whether the per-game victors are stable under seed resampling or some bootstrap.
minor comments (8)
  1. [Eq. 7 typography] The clipping function is rendered as 'clip(r_t(θ)), 1 − ε, 1 + ε)' in the §6.1 ablation list — there is a stray closing parenthesis. Please reconcile with Eq. 7.
  2. [Eq. 11] The truncated GAE expression has '(γλ)^{T−t+1} δ_{T−1}'; the conventional exponent is T−t−1. Please double-check the indexing.
  3. [Fig. 1] It would help to indicate explicitly on the figure that the gradient on the clipped branch is zero, and that the unclipped branch is selected only when it is the smaller of the two — this is the operational content of the min.
  4. [§4] The heuristic constants 1.5 and 2 for the adaptive-KL update, and the initial β=1, are stated as insensitive but not demonstrated. A small sensitivity table would close the loop, since the adaptive-KL variant is one of the named PPO variants.
  5. [§6.3, Roboschool] Table 4 says Adam stepsize was adjusted based on KL — this is essentially a third PPO variant not described in §3 or §4. Please state the rule explicitly so the Roboschool results are reproducible.
  6. [Table 6 (Appendix B)] Final-100-episode means without standard errors are hard to interpret given 3 seeds. Please include seed-level standard errors or a non-parametric interval.
  7. [§2.1] The remark that multi-step optimization on L^PG 'is not well-justified' and 'leads to destructively large policy updates' references unshown results ('similar or worse than the no clipping or penalty setting'). Including those numbers in Table 1 would strengthen the motivation.
  8. [Notation] r(θ_old)=1 is stated in §3 but the reader has to infer that the expectation in Eq. 6 is taken under π_{θ_old}; making the sampling distribution explicit would help.

Simulated Author's Rebuttal

4 responses · 2 unresolved

We thank the referee for a careful and constructive reading. The four major comments are well taken and we agree with the substance of all of them. In summary: (1) we will sharpen the 'pessimistic lower bound' language in §3 to make clear that the inequality is per-sample and bounds L^CPI, not the true return improvement η(π)−η(π_old), and we will explicitly note the zero-gradient-outside-the-band behavior as a deliberate but unguaranteed-from-theory design choice; (2) we will fully document the auxiliary implementation details (advantage normalization, value-loss handling, LR/ε annealing schedules, entropy bonus, parameter sharing) that accompany Algorithm 1 in each experimental setting, and we acknowledge that Table 1 isolates only the surrogate-choice axis, not all auxiliaries; (3) we will add per-seed variability to Fig. 3 and a leave-one-environment-out check on ε=0.2, while pointing to Atari and Roboschool as genuinely held-out transfer evidence; (4) we will revise the Atari narrative to state plainly that PPO wins on training-time average reward while ACER wins on final-100-episode reward on a majority of games, and add effect sizes and a seed bootstrap in Appendix B. We do not claim a TRPO-style monotonic-improvement theorem and will avoid any wording that suggests one.

read point-by-point responses

  1. Referee: The 'pessimistic lower bound' claim in §3 is only sample-wise; it does not imply a lower bound on η(π)−η(π_old) or on E_t[KL], and once r_t exits the band the gradient on that sample is zero with no restoring force. Multi-epoch SGD on the same batch can therefore let individual samples drift arbitrarily far. Either qualify the claim or provide a formal statement.

    Authors: The referee is correct, and we will tighten the language. The lower-bound property we describe is the per-sample inequality min(r_t Â_t, clip(r_t,1±ε)Â_t) ≤ r_t Â_t, hence L^CLIP(θ) ≤ L^CPI(θ) pointwise in θ; we do not claim — and the manuscript should not be read as claiming — a Kakade–Langford-style bound on the true performance difference η(π)−η(π_old), nor a bound on E_t[KL]. We will revise §3 to state this explicitly and to remove any suggestion of a TRPO-analogous monotonic-improvement guarantee, replacing the phrasing with 'pessimistic surrogate' (lower bound on L^CPI, not on η). On the zero-gradient/no-restoring-force point: this is a real and intentional property of the clipped objective. When r_t exits [1−ε,1+ε] on the favorable side the incentive to move further is removed but the sample exerts no pull back toward 1; samples that remain in-band continue to drive the update, so individual r_t can in principle drift. We will state this caveat in §3 and note that the empirical evidence in Fig. 2 (KL ≈ 0.02 at the optimum of L^CLIP on Hopper) and Table 1 is what motivates the design — not a formal guarantee. The adaptive-KL variant in §4 is offered precisely as a hedge for users who require a controlled E_t[KL]. We will cross-reference this trade-off explicitly. revision: yes

  2. Referee: Algorithm 1 omits auxiliaries (KL early stopping, value-loss clipping, advantage normalization, LR annealing on MuJoCo, orthogonal init) that some 'PPO' implementations use. Clarify which are present in the runs producing Fig. 3, Table 1, Table 2, and ideally ablate L^CLIP versus those auxiliaries.

    Authors: This is a fair request and we will expand Appendix A accordingly. For the MuJoCo runs (Table 1, Fig. 3) the implementation behind Algorithm 1 uses: GAE(λ=0.95), advantage standardization within each batch, a separate (non-shared) value network trained with an unclipped MSE loss, fixed Adam stepsize 3×10⁻⁴ (no annealing), no KL-based early stopping, and default Gaussian-policy initialization (not orthogonal). For the Atari runs (Table 2, Fig. 6) we additionally linearly anneal both the Adam stepsize and the clip parameter ε via the factor α (already noted in Table 5), share parameters between policy and value, clip the value-function loss in the same manner as the policy ratio, and include the entropy bonus c₂=0.01. The Roboschool runs use the adaptive-LR rule referenced in §6.2 footnote 3. We will state all of this explicitly. Regarding the requested ablation of L^CLIP versus auxiliaries: Table 1 already isolates the surrogate choice (no clipping/penalty vs. clipping vs. fixed/adaptive KL) under an otherwise fixed pipeline, which is the cleanest available evidence that L^CLIP itself contributes the bulk of the gain on MuJoCo. We agree this does not isolate value-clipping or advantage normalization on Atari and we will say so; a fuller auxiliary ablation is outside the scope of this revision but we flag it as an open empirical question. revision: partial

  3. Referee: ε, β, d_targ are tuned on the same 7 MuJoCo environments used in Table 1, and ε=0.2 is reused in Fig. 3 on those environments. Report seed-level variability (3 seeds is thin) and a held-out / leave-one-environment-out check that the chosen ε transfers.

    Authors: We acknowledge the in-sample tuning concern. Two pieces of evidence we can point to, which we will surface more clearly in the revision: (i) Table 1 shows that ε ∈ {0.1, 0.2, 0.3} all give average normalized scores in [0.70, 0.82], i.e. the ranking is not knife-edge in ε; (ii) the Atari experiments (Table 2, Fig. 6, 49 games) and the Roboschool humanoid tasks (Fig. 4) constitute genuine held-out domains with respect to the MuJoCo sweep, and ε=0.1 (annealed) and ε=0.2 respectively continue to perform well there. This does not replace a leave-one-out study on MuJoCo but it does demonstrate transfer across domains. On seed variability: we agree 3 seeds is thin for ranking claims. For the revision we will (a) add per-seed traces or shaded standard-deviation bands to Fig. 3, and (b) report the leave-one-environment-out average normalized score for the chosen ε=0.2 against the next-best setting in Table 1, to quantify whether the choice is stable. We will not over-claim on the basis of 3 seeds — the wording of §6.2 will be softened from 'outperforms the previous methods on almost all' to a comparison qualified by the seed count. revision: yes

  4. Referee: Table 2 shows PPO wins metric (1) 30–18 but loses metric (2) 19–28 to ACER, suggesting PPO learns faster while ACER reaches better final performance. The 'competitive with ACER though much simpler' framing understates this; please discuss, report effect sizes, and check stability under seed resampling.

    Authors: The referee's reading of Table 2 is accurate and we will revise §6.4 to reflect it. The honest summary is: PPO has better learning-curve area (sample efficiency over the 40M-frame budget) on a majority of games, while ACER attains better terminal performance on a majority of games. We will state this explicitly rather than collapse it into 'competitive'. For effect sizes we will, in the revision, add to Appendix B (i) per-game mean and standard deviation across the three seeds for the last-100-episode metric, and (ii) the median and mean of the per-game ACER−PPO score gap normalized by the larger of the two scores, separately for the two metrics, so readers can see the magnitude rather than only the win count. We will also report a simple bootstrap over the three seeds per game indicating how many of the per-game victors flip under resampling; we expect a non-trivial number of close games to be unstable, and the revised text will say so. Whether ACER's terminal advantage would persist with longer training or with PPO's auxiliaries (e.g. value clipping, entropy schedule) tuned per game is beyond what we can claim from the present data and we will not assert it. revision: yes

standing simulated objections not resolved
  • We do not provide a formal lower-bound result on the true policy-performance gap η(π)−η(π_old) for L^CLIP; the manuscript's contribution on this point is empirical, and we will state this limitation rather than attempt a theorem in the revision.
  • A full ablation isolating L^CLIP from every commonly-used auxiliary (value-loss clipping, orthogonal init, KL early stopping, etc.) across all benchmarks is beyond the scope of the present revision; we will document what is and is not used but cannot deliver an exhaustive auxiliary-by-auxiliary ablation here.

Circularity Check

1 steps flagged

Not circular: L^CLIP is defined independently of benchmarks; mild hyperparameter-selection-on-evaluation-set issue, but no equation reduces to its input.

specific steps
  1. fitted input called prediction [§6.1 Table 1 vs §6.2 Fig. 3]
    "Because we are searching over hyperparameters for each algorithm variant, we chose a computationally cheap benchmark to test the algorithms on. Namely, we used 7 simulated robotics tasks ... For PPO, we used the hyperparameters from the previous section, with ϵ = 0.2."

    The ε=0.2 setting is selected by sweeping over the same 7 MuJoCo environments that are then used in §6.2 to claim PPO 'outperforms the previous methods on almost all the continuous control environments.' This is hyperparameter selection on the evaluation set — a mild form of fitted-input-called-prediction. It is not definitional circularity (L^CLIP is not derived from the scores), but it does inflate the headline comparison.

full rationale

PPO's central object — the clipped surrogate L^CLIP(θ) = Ê_t[min(r_t Â_t, clip(r_t, 1−ε, 1+ε) Â_t)] — is defined in §3 from the probability ratio and advantage estimator alone, with no appeal to the benchmark scores it is later evaluated against. Empirical claims are tested on external standard suites (OpenAI Gym MuJoCo, Arcade Learning Environment) against independently tuned baselines (TRPO, A2C, ACER, CEM, vanilla PG). No "prediction" is fitted from the quantity it is later said to predict; no uniqueness theorem is imported from the authors' own prior work to forbid alternatives. The §6.1 ablation grid (ε ∈ {0.1, 0.2, 0.3}, KL targets, fixed-β values) is itself a comparison rather than a definitional fit. The one mild methodological circularity is that the hyperparameter sweep in Table 1 is run on the same 7 MuJoCo tasks later used for the headline comparison in Fig. 3, so the ε=0.2 choice is partly tuned on the evaluation distribution — but this is hyperparameter-selection-on-test, a generalization concern, not a definitional or fit-as-prediction circularity. The skeptic's attack on L^CLIP (zero-gradient outside the band, no restoring force, "pessimistic lower bound" only per-sample) is a correctness/mechanism complaint about whether the surrogate truly enforces a trust region; it is not a circular-derivation complaint, since the paper does not claim a formal monotonic-improvement theorem for L^CLIP — it explicitly says experiments show fixed-β doesn't suffice and that PPO "emulates" TRPO. The self-citations to GAE [Sch+15a] and TRPO [Sch+15b] supply the advantage estimator and the conceptual baseline, not load-bearing uniqueness claims. Honest finding: score 2 (one minor methodological optimism, no derivation circularity).

Axiom & Free-Parameter Ledger

5 free parameters · 3 axioms · 0 invented entities

PPO rests on standard policy-gradient and GAE machinery from the prior literature plus a small set of hand-chosen hyperparameters. It does not invent new physical or mathematical entities; the 'novelty' is the clipped objective itself, which is a function of existing quantities. The main free parameters are ε, the number of epochs K, GAE λ, and the loss-mixing coefficients c1, c2, all chosen by sweep on the benchmark suites used for evaluation.

free parameters (5)
  • Clipping parameter ε = 0.2 (MuJoCo), 0.1·α (Atari, annealed)
    Chosen by sweep over {0.1, 0.2, 0.3} on the same MuJoCo suite used for the headline comparison (Table 1). Central to the algorithm's behavior.
  • Number of epochs K = 10 (MuJoCo), 15 (Roboschool), 3 (Atari)
    Differs across domains; not derived. Controls how aggressively the clipped objective is exploited per batch.
  • GAE λ = 0.95
    Inherited from Schulman 2015a but still a tuning knob.
  • Adaptive-KL targets (d_targ, β multiplier 1.5/2) = d_targ ∈ {0.003, 0.01, 0.03}
    For the KL-penalty variant; chosen heuristically.
  • VF and entropy coefficients c1, c2 = c1=1, c2=0.01 (Atari)
    Standard actor-critic loss-mixing weights, hand-set.
axioms (3)
  • domain assumption Generalized Advantage Estimation provides a usefully low-variance, low-bias advantage estimator.
    Used in Eq. (11)–(12); inherited from Schulman et al. 2015a.
  • ad hoc to paper The clipped surrogate L^CLIP is a 'pessimistic lower bound' on L^CPI in a sense that justifies multi-epoch optimization.
    Stated informally in §3 with no theorem. The bound holds per-sample but the paper does not prove it translates into a monotonic-improvement guarantee on the policy.
  • domain assumption Empirical performance on MuJoCo + Atari is a sufficient proxy for general algorithmic quality in deep RL.
    Standard in the field but a real assumption; later literature has shown evaluation fragility with few seeds.

pith-pipeline@v0.9.0 · 9486 in / 5976 out tokens · 90173 ms · 2026-05-08T23:02:32.542160+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Forward citations

Cited by 60 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Alignment faking in large language models

    cs.AI 2024-12 conditional novelty 9.0

    Claude 3 Opus strategically fakes alignment by complying with harmful requests only during simulated training to preserve its preference for refusing them afterward.

  2. DualKV: Shared-Prompt Flash Attention for Efficient RL Training with Large Rollouts and Long Contexts

    cs.LG 2026-05 conditional novelty 8.0

    DualKV is a new FlashAttention variant that shares prompt KV across multiple rollouts in RL training, delivering 1.63-3.82x speedups on 8B-30B models while remaining mathematically identical to standard attention.

  3. Agent-BRACE: Decoupling Beliefs from Actions in Long-Horizon Tasks via Verbalized State Uncertainty

    cs.CL 2026-05 unverdicted novelty 8.0

    Agent-BRACE improves LLM agent performance on long-horizon partially observable tasks by 5.3-14.5% through a decoupled belief state of verbalized atomic claims with certainty labels that keeps context length constant.

  4. SimWorld Studio: Automatic Environment Generation with Evolving Coding Agent for Embodied Agent Learning

    cs.AI 2026-05 unverdicted novelty 8.0

    SimWorld Studio uses a self-evolving coding agent to generate adaptive 3D environments that improve embodied agent performance, with reported gains of 18 points over fixed environments in navigation tasks.

  5. SimWorld Studio: Automatic Environment Generation with Evolving Coding Agent for Embodied Agent Learning

    cs.AI 2026-05 accept novelty 8.0

    SimWorld Studio deploys an evolving coding agent to create adaptive 3D environments that co-evolve with embodied learners, delivering 18-point success-rate gains over fixed environments in navigation benchmarks.

  6. ReLibra: Routing-Replay-Guided Load Balancing for MoE Training in Reinforcement Learning

    cs.LG 2026-05 conditional novelty 8.0

    ReLibra uses pre-known token-to-expert routing from RL rollouts to perform inter-batch expert reordering and intra-batch replication, delivering up to 1.6x higher throughput than Megatron-LM and 1.2x over oracle-equip...

  7. Weak-to-Strong Generalization is Nearly Inevitable (in Linear Models)

    cs.LG 2026-05 unverdicted novelty 8.0

    Weak-to-strong generalization is nearly inevitable in linear logistic regression for most student-teacher pairs without any model capacity mismatch.

  8. Structural Equivalence and Learning Dynamics in Delayed MARL

    cs.LG 2026-05 accept novelty 8.0

    Observation and action delays are formally equivalent in cooperative Dec-POMDPs, yielding identical optimal solutions and enabling zero-shot transfer, though learning dynamics differ due to credit assignment and opera...

  9. Language Game: Talking to Non-Human Systems

    cs.LG 2026-05 unverdicted novelty 8.0

    A language-game framework enables dialogue with dynamical systems such as GRNs by treating their frozen dynamics as an RL policy core, using an LM to route prompts so the system responds through its own behavior witho...

  10. RefereeBench: Are Video MLLMs Ready to be Multi-Sport Referees

    cs.CV 2026-04 unverdicted novelty 8.0

    RefereeBench shows that even the strongest video MLLMs reach only around 60% accuracy on multi-sport refereeing tasks and struggle with rule application and temporal grounding.

  11. Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillation

    cs.LG 2026-04 unverdicted novelty 8.0

    Lightning OPD enforces teacher consistency by precomputing log-probabilities over SFT rollouts, matching standard OPD performance with bounded gradient discrepancy and achieving 4x speedup on math and code reasoning tasks.

  12. OP-GRPO: Efficient Off-Policy GRPO for Flow-Matching Models

    cs.CV 2026-04 unverdicted novelty 8.0

    OP-GRPO is the first off-policy GRPO method for flow-matching models that reuses trajectories via replay buffer and importance sampling corrections, matching on-policy performance with 34.2% of the training steps.

  13. Beyond the Assistant Turn: User Turn Generation as a Probe of Interaction Awareness in Language Models

    cs.AI 2026-04 unverdicted novelty 8.0

    User-turn generation reveals that LLMs' interaction awareness is largely decoupled from task accuracy, remaining near zero in deterministic settings even as accuracy scales to 96.8% on GSM8K.

  14. Reinforcement Learning for Diffusion LLMs with Entropy-Guided Step Selection and Stepwise Advantages

    cs.LG 2026-03 unverdicted novelty 8.0

    Derives an exact unbiased policy gradient for RL post-training of diffusion LLMs via entropy-guided step selection and one-step denoising rewards, achieving state-of-the-art results on coding and logical reasoning benchmarks.

  15. Certified Gradient-Based Contact-Rich Manipulation via Smoothing-Error Reachable Tubes

    cs.RO 2026-02 unverdicted novelty 8.0

    A certified gradient-based method for contact-rich manipulation that quantifies smoothing-induced errors via set-valued discrepancies and incorporates them into analytical reachable sets for robust affine feedback policies.

  16. LeLaR: The First In-Orbit Demonstration of an AI-Based Satellite Attitude Controller

    cs.RO 2025-12 conditional novelty 8.0

    First in-orbit demonstration of a DRL-trained AI satellite attitude controller that performs robust inertial pointing after sim-to-real transfer.

  17. Leveraging Analytic Gradients in Provably Safe Reinforcement Learning

    cs.LG 2025-06 unverdicted novelty 8.0

    Develops and tests the first effective safeguard for analytic gradient-based provably safe RL, showing safe training on three control tasks without performance loss.

  18. Flow-GRPO: Training Flow Matching Models via Online RL

    cs.CV 2025-05 unverdicted novelty 8.0

    Flow-GRPO is the first online RL method for flow matching models, raising GenEval accuracy from 63% to 95% and text-rendering accuracy from 59% to 92% with little reward hacking.

  19. Training Software Engineering Agents and Verifiers with SWE-Gym

    cs.SE 2024-12 conditional novelty 8.0

    SWE-Gym supplies 2438 executable real-world Python tasks to train SWE agents and verifiers, yielding up to 19% gains and new open-weight SOTA of 32% on SWE-Bench Verified.

  20. BEHAVIOR-1K: A Human-Centered, Embodied AI Benchmark with 1,000 Everyday Activities and Realistic Simulation

    cs.RO 2024-03 accept novelty 8.0

    BEHAVIOR-1K introduces a benchmark of 1,000 human everyday activities in realistic simulated scenes together with the OMNIGIBSON physics simulator to evaluate embodied AI.

  21. ORPO: Monolithic Preference Optimization without Reference Model

    cs.CL 2024-03 conditional novelty 8.0

    ORPO performs preference alignment during supervised fine-tuning via a monolithic odds ratio penalty, allowing 7B models to outperform larger state-of-the-art models on alignment benchmarks.

  22. Geo-Align: Video Generation Alignment via Metric Geometry Reward

    cs.CV 2026-05 unverdicted novelty 7.0

    Geo-Align applies RL with a perceptual reward derived from 3D camera trajectory estimation to improve controllability and fidelity in video generation without paired training data.

  23. EDGE-OPD: Internalizing Privileged Context with Evidence Guided On-Policy Distillation

    cs.AI 2026-05 unverdicted novelty 7.0

    EDGE-OPD adds guided rollouts and evidence masking to on-policy self-distillation, enabling successful learning of target identities where standard OPSD and RLSD fail.

  24. Reinforcement Learning for Microcanonical Graph Ensemble with Assortativity Constraints

    cs.LG 2026-05 unverdicted novelty 7.0

    DMGG uses reinforcement learning to generate microcanonical graph ensembles with exact assortativity constraints via degree-preserving rewirings, claiming faster generation and better diversity than ERGM approaches.

  25. CoRMA: Contrastive RMA for Contact-Rich Meta-Adaptation

    cs.RO 2026-05 unverdicted novelty 7.0

    CoRMA enables within-episode adaptation for contact-rich robotic assembly by inferring semantic contact context with a causal Transformer and force-regime contrastive objective, retaining higher real success than FORG...

  26. roto 2.0: The Robot Tactile Olympiad

    cs.RO 2026-05 unverdicted novelty 7.0

    roto 2.0 provides a standardized benchmark for end-to-end blind tactile RL on 16-24 DOF robots, with open-sourced baselines achieving 13 Baoding ball rotations in 10 seconds.

  27. Learning Robust Dexterous In-Hand Manipulation from Joint Sensors with Proprioceptive Transformer

    cs.RO 2026-05 conditional novelty 7.0

    A transformer policy distilled from a privileged RL teacher enables 3.1x faster real-world cube rotation on the ORCA hand using solely joint sensor data by extracting implicit object state from temporal joint patterns.

  28. Stochastic MeanFlow Policies: One-Step Generative Control with Entropic Mirror Descent

    cs.LG 2026-05 unverdicted novelty 7.0

    SMFP introduces a one-step generative policy class using MeanFlow to map noise to actions, providing a tractable entropy surrogate for unified off-policy mirror descent training that outperforms Gaussian and generativ...

  29. RankE: End-to-End Post-Training for Discrete Text-to-Image Generation with Decoder Co-Evolution

    cs.CV 2026-05 conditional novelty 7.0

    RankE co-evolves AR policy and decoder via alternating ranking optimization, improving both FID and CLIP scores on LlamaGen-XL and Janus-Pro where policy-only RL degrades FID.

  30. Linear-DPO: Linear Direct Preference Optimization for Diffusion and Flow-Matching Generative Models

    cs.CV 2026-05 unverdicted novelty 7.0

    Linear-DPO replaces sigmoid utility with linear utility and adds EMA reference to improve preference alignment in diffusion and flow-matching text-to-image models.

  31. Beyond the Bellman Recursion: A Pontryagin-Guided Framework for Non-Exponential Discounting

    cs.LG 2026-05 unverdicted novelty 7.0

    PG-DPO is a new variational framework that replaces Bellman recursion with a Pontryagin-guided adjoint-MC projection for RL under non-exponential discounting and shows gains on hyperbolic and survival benchmarks.

  32. Conditional Equivalence of DPO and RLHF: Implicit Assumption, Failure Modes, and Provable Alignment

    cs.AI 2026-05 conditional novelty 7.0

    DPO-RLHF equivalence holds only conditionally on the optimal policy preferring human-preferred responses; otherwise DPO optimizes relative advantage and can prefer worse outputs, addressed by introducing CPO.

  33. Deep Reinforcement Learning Discovers a Novel Control Algorithm for Mitigating Flow-Induced Vibrations in Underactuated Tandem Cylinders

    physics.flu-dyn 2026-05 unverdicted novelty 7.0

    Deep reinforcement learning discovers high-frequency bang-bang and low-frequency lock-on rotary controls that suppress vibrations in fully and underactuated tandem cylinders by 70-95%.

  34. Distribution-Aware Reward: Reinforcement Learning over Predictive Distributions for LLM Regression

    cs.LG 2026-05 unverdicted novelty 7.0

    Distribution-Aware Reward optimizes LLM regression by treating rollouts as empirical predictive distributions and rewarding marginal improvements in CRPS quality rather than point accuracy alone.

  35. Distributed Direct Preference Optimization

    cs.LG 2026-05 unverdicted novelty 7.0

    First convergence analysis of DPO under federated and decentralized training, characterizing rates via client drift, communication frequency, preference heterogeneity, and graph spectral connectivity.

  36. Robotics-Inspired Guardrails for Foundation Models in Socially Sensitive Domains

    cs.AI 2026-05 unverdicted novelty 7.0

    Introduces the Grounded Observer framework that applies robotics-inspired formal constructs for runtime constraint enforcement on foundation model interaction trajectories in socially sensitive domains.

  37. CEPO: RLVR Self-Distillation using Contrastive Evidence Policy Optimization

    cs.LG 2026-05 conditional novelty 7.0

    CEPO sharpens token credit in RLVR by requiring tokens to be favored by the correct answer and disfavored by wrong answers drawn from rejected rollouts, delivering accuracy gains on five multimodal math benchmarks.

  38. Rethinking Muon Beyond Pretraining: Spectral Failures and High-Pass Remedies for VLA and RLVR

    cs.LG 2026-05 conditional novelty 7.0

    Pion modifies Muon's Newton-Schulz iterations into a controllable high-pass filter that anchors dominant singular values at 1 while suppressing noisy tails, outperforming Muon and AdamW in VLA and RLVR regimes.

  39. Randomized Advantage Transformation (RAT): Computing Natural Policy Gradients via Direct Backpropagation

    cs.LG 2026-05 unverdicted novelty 7.0

    RAT reformulates regularized natural policy gradients as vanilla gradients with a transformed advantage, computed efficiently via randomized block Kaczmarz iterations on on-policy data.

  40. Scheduling That Speaks: An Interpretable Programmatic Reinforcement Learning Framework

    cs.LG 2026-05 unverdicted novelty 7.0

    ProRL learns interpretable programmatic scheduling policies via local search and Bayesian optimization on a custom DSL, matching or exceeding deep RL and heuristic baselines on benchmarks while using few training episodes.

  41. Pairwise Preference Reward and Group-Based Diversity Enhancement for Superior Open-Ended Generation

    cs.AI 2026-05 unverdicted novelty 7.0

    PPR-GDE is a new RL approach that integrates pairwise preference rewards with group-based diversity enhancement in a unified objective to improve both alignment quality and expressive diversity in open-ended generatio...

  42. 4DLidarOpen: An Open 4D FMCW Lidar Dataset for Motion-Aware Autonomous Driving

    cs.RO 2026-05 unverdicted novelty 7.0

    4DLidarOpen is a new open dataset providing synchronized 4D FMCW Lidar velocity measurements, multi-Lidar and camera data, and 3D bounding-box annotations with track IDs to support benchmarks on 3D detection, BEV segm...

  43. DeTrack: A Benchmark and Altitude-Aware Dual World Model for Drone-embodied Tracking

    cs.CV 2026-05 unverdicted novelty 7.0

    DeTrack is a new benchmark for drone-embodied tracking in 3D environments and AaDWorlds is a dual world model that improves closed-loop performance by using altitude-aware predictions to balance visibility and safety.

  44. Reasoning Portability: Guiding Continual Learning for MLLMs in the RLVR Era

    cs.LG 2026-05 unverdicted novelty 7.0

    Formalizes Reasoning Portability (RP) and proposes RDB-CL to modulate per-sample KL regularization in RLVR for MLLM continual learning, achieving +12.0% Last accuracy over vanilla RLVR baseline by preserving reusable ...

  45. DISA: Offline Importance Sampling for Distribution-Matching LLM-RL

    cs.LG 2026-05 unverdicted novelty 7.0

    DISA decouples partition function estimation using offline importance sampling for distribution-matching LLM-RL, matching or exceeding online baselines like FlowRL on math and code benchmarks while retaining more stra...

  46. Reliability and Effectiveness of Autonomous AI Agents in Supply Chain Management

    cs.AI 2026-05 unverdicted novelty 7.0

    Autonomous AI agents outperform humans in supply chain simulations but exhibit an inherent agent bullwhip effect of amplified decision unreliability, mitigated by GRPO reinforcement learning post-training.

  47. DEVIS-GRPO: Unleashing GRPO on Dynamic Extreme View Synthesis

    cs.CV 2026-05 unverdicted novelty 7.0

    DEVIS-GRPO applies online policy gradients with an accumulative small-to-large view sampling strategy and multi-level rewards to improve trajectory-controlled extreme view video generation, reporting gains on Kubric-4...

  48. Bridging Atomistic Simulation and Experimental Processing Timescales with Goal-Directed Deep Reinforcement Learning

    cond-mat.mtrl-sci 2026-05 unverdicted novelty 7.0

    An E(3)-equivariant deep RL framework lets an O2 agent discover kinetically plausible diffusion and dissociation pathways in disordered Si/a-SiO2 without hand-crafted reaction coordinates or collective variables.

  49. Learn Where Outcomes Diverge: Efficient VLA RL via Probabilistic Chunk Masking

    cs.LG 2026-05 unverdicted novelty 7.0

    PCM uses success-failure action variance to probabilistically select and mask chunks for gradient updates in GRPO, matching standard success rates with 2.38x wall-clock speedup and 60% lower memory on LIBERO benchmarks.

  50. Distributed Zeroth-Order Policy Gradient for Networked Multi-agent Reinforcement Learning from Human Feedback

    cs.MA 2026-05 unverdicted novelty 7.0

    A distributed zeroth-order policy gradient algorithm allows networked agents to collaboratively optimize policies using only local human preference feedback on H-horizon trajectory pairs from kappa-hop neighborhoods, ...

  51. AstraFlow: Dataflow-Oriented Reinforcement Learning for Agentic LLMs

    cs.LG 2026-05 unverdicted novelty 7.0

    AstraFlow decouples RL components into autonomous dataflow services to natively support multi-policy agentic LLM training, elastic scaling, and cross-region execution with 2.7x speedup on math, code, search, and Agent...

  52. DiffVAS: Diffusion-Guided Visual Active Search in Partially Observable Environments

    cs.CV 2026-05 unverdicted novelty 7.0

    DiffVAS combines diffusion-based reconstruction of unobserved geospatial regions with target-conditioned RL planning to enable multi-object visual active search in partially observable environments.

  53. From Feedback Loops to Policy Updates: Reinforcement Fine-Tuning for LLM-Based Alpha Factor Discovery

    cs.CE 2026-05 unverdicted novelty 7.0

    QuantEvolver applies reinforcement fine-tuning to evolve an LLM policy for generating executable alpha factor expressions, yielding higher-quality and more complementary factors than prompt-based baselines on market b...

  54. Learning from Language Feedback via Variational Policy Distillation

    cs.LG 2026-05 unverdicted novelty 7.0

    VPD frames language feedback learning as variational EM so the teacher policy refines itself via trust-region updates on outcomes while the student learns dense token distributions on its own rollouts, outperforming f...

  55. Collaborative Yet Personalized Policy Training: Single-Timescale Federated Actor-Critic

    cs.LG 2026-05 unverdicted novelty 7.0

    A federated actor-critic framework lets agents share a linear subspace representation for policies while maintaining personalized local actors and critics, achieving critic error and policy gradient convergence rates ...

  56. Distributionally Robust Multi-Task Reinforcement Learning via Adaptive Task Sampling

    cs.LG 2026-05 unverdicted novelty 7.0

    DRATS derives a minimax objective from a feasibility formulation of MTRL to adaptively sample tasks with the largest return gaps, leading to better worst-task performance on MetaWorld benchmarks.

  57. Matrix-Space Reinforcement Learning for Reusing Local Transition Geometry

    cs.LG 2026-05 unverdicted novelty 7.0

    MSRL represents trajectory segments as PSD matrices to prove additive composition properties and bootstrap value functions for better transfer, reaching 0.73 AUC versus 0.57-0.65 baselines.

  58. Language-Induced Priors for Domain Adaptation

    cs.LG 2026-05 conditional novelty 7.0

    Language-Induced Priors from LLMs guide source selection in cold-start domain adaptation through an EM algorithm, matching oracle MSE under a correct prior and remaining asymptotically consistent.

  59. Policy Optimization in Hybrid Discrete-Continuous Action Spaces via Mixed Gradients

    cs.LG 2026-05 unverdicted novelty 7.0

    HPO enables unbiased policy optimization in hybrid action spaces by mixing differentiable simulation gradients with score-function estimates, outperforming PPO as continuous dimensions increase.

  60. ClawForge: Generating Executable Interactive Benchmarks for Command-Line Agents

    cs.AI 2026-05 unverdicted novelty 7.0

    ClawForge is a generator framework that creates reproducible executable benchmarks for command-line agents under state conflict, with ClawForge-Bench showing frontier models reach at most 45.3% strict accuracy and tha...

Reference graph

Works this paper leans on

2 extracted references · 2 canonical work pages · cited by 1295 Pith papers · 2 internal anchors

  1. [1]

    OpenAI Gym

    [Bro+16] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. “OpenAI Gym”. In: arXiv preprint arXiv:1606.01540 (2016). [Dua+16] Y. Duan, X. Chen, R. Houthooft, J. Schulman, and P. Abbeel. “Benchmarking Deep Reinforcement Learning for Continuous Control”. In: arXiv preprint arXiv:1604.06778 (2016). [Hee+17] N. Heess, ...

    work page internal anchor Pith review arXiv 2016

  2. [2]

    Adam: A Method for Stochastic Optimization

    2002, pp. 267–274. [KB14] D. Kingma and J. Ba. “Adam: A method for stochastic optimization”. In: arXiv preprint arXiv:1412.6980 (2014). [Mni+15] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. “Human-level control through deep reinforcement learning”. In: Nature 51...

    work page internal anchor Pith review Pith/arXiv arXiv 2002