Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

On the Effect of Regularization in Policy Mirror Descent

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Policy Mirror Descent's two regularizers partially substitute for each other, but their exact pairing controls robustness to temperature settings.

desk verdict Large-scale empirical mapping of PMD regularization interplay; plausible main claims, but the robustness ranking and scaling law need grid-sensitivity checks and code/data release before the quantitative details can be trusted. read the letter →

arxiv 2507.08718 v1 pith:QUJY3DGG submitted 2025-07-11 cs.LG

Jan Felix Kleuker , Aske Plaat , Thomas Moerland This is my paper
classification cs.LG
keywords policymirrordescentMDPregularizationdrifttemperaturerobustnessentropyhyperparametersensitivityreinforcementlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Policy Mirror Descent (PMD) couples an MDP regularizer that reshapes the reward signal with a Drift regularizer that keeps successive policies close. This paper tests, across over 500,000 training seeds on four small reinforcement-learning environments, how these two regularizers interact when their strengths (temperatures $\alpha$ and $\lambda$) are varied. The central finding is that the two can partially stand in for each other—the well-performing region of temperatures is L-shaped, so a low setting of one can be offset by raising the other—but the precise pair of regularizer functions matters for how robust the algorithm is to bad temperature choices. The results also show that the Drift temperature should stay constant rather than be annealed, and that both temperatures must grow roughly linearly with the scale of environment rewards.

What carries the argument

The central object is the PMD policy-improvement objective that couples the two regularizers through their temperatures: $\alpha h(\pi(\cdot|s))$ for the MDP regularizer and $\lambda_k D(\pi,\pi_k|s)$ for the Drift regularizer. The argument's main diagnostic is the heat map of mean normalized return over the $29\times29$ grid of $(\alpha,\lambda)$ values; the reproducible L-shaped arrangement of high-performing cells is what the authors read as evidence of substitutability, and the normalized area under the performance-frequency curve, $Rbst_T$, is the device that turns the heat maps into a robustness ranking of regularizer pairs.

What would settle it

To test the ranking claim: rerun the robustness measure on a denser and wider temperature grid (for example, 100×100 values spanning the same ranges plus one decade beyond) for the eleven $(h, D)$ pairs; if the order in Table 1 changes materially, the ranking is an artifact of grid choice. To test the scaling claim: extract minimal temperatures using a success threshold of 0.75 instead of 0.85; if the slopes of the fitted lines change significantly, the linear relationship does not hold as stated.

Watch

Extended reading notes

Core claim

On the paper's own terms, the contribution is an empirical mapping of the $(\alpha, \lambda)$ temperature plane for the PMD objective $$\min_\$\theta$ \mathbb{E}_{s\sim D}\big[\mathbb{E}_{a\sim\pi_\$\theta$}[\,{-}Q(s,a)] + \$\alpha$\, h(\pi_\$\theta$(\cdot|s)) + \$\lambda$\, D(\pi_\$\theta$;\pi_{\theta_k}|s)\big],$$ across eleven combinations of MDP regularizer $h$ (negative Shannon entropy, Tsallis entropy, squared norm, max) and Drift regularizer $D$ (KL divergence and matching Bregman divergences). The paper claims that the two regularizers can partially substitute for each other, since every heat map shows an L-shaped region of temperature pairs that achieve high return, and that robustness to temperature selection depends on the whole pair $(h,D)$, not on either regularizer alone. It further claims that keeping the Drift temperature $\lambda$ constant outperforms annealing it, and that the minimal temperatures needed for successful learning grow linearly with the maximum return of the environment.

Load-bearing premise

The robustness ranking and the linear scaling laws rest on the assumption that the hand-picked 29×29 grid of temperature values, and the post hoc restriction of the minimal-temperature extraction to a 'sensible region', fairly represent each algorithm's full behavior; a different grid range or density, or a different success threshold, could change the ordering and the fitted lines.

Editorial extensions

If this is right

  • When tuning PMD-style algorithms, researchers can compensate a small MDP-regularization temperature with a larger Drift temperature, and vice versa, without necessarily losing performance.
  • Robustness to hyperparameter choice is a property of the $(h, D)$ combination; swapping either regularizer can collapse or greatly expand the set of acceptable temperatures.
  • Adaptive or annealed schedules for the Drift temperature are not universally beneficial; constant $\lambda$ matches or beats them in the settings tested.
  • The temperature values that work best are not transferable across environments with different reward scales, because both $\alpha$ and $\lambda$ scale roughly linearly with maximum return.
  • Even a theoretically valid regularizer choice can perform badly if paired with an incompatible partner, so regularizer selection must be treated jointly.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the linear temperature–reward-scaling finding generalizes, it suggests a practical recipe: set initial temperatures proportional to an estimate of the return range, rather than re-tuning from scratch for each new environment.
  • The L-shaped substitutability pattern hints at a deeper equivalence class: what matters may be the total effective regularization strength along some diagonal of $(\alpha, \lambda)$, rather than the two strengths separately; a direct test would be to fit a single combined regularization measure and see whether performance collapses onto it.
  • The finding that the non-smooth max regularizer pairs most robustly with its Bregman divergence challenges the field's focus on smooth entropy-based choices and invites a broader search over non-standard regularizer families.
  • The robustness metric $Rbst$ depends on the chosen grid, so a natural extension is to compute, for each pair, the volume of the 'good' region in a continuous temperature space, which would remove the grid-density dependence of the ranking.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper presents a large-scale empirical study of Policy Mirror Descent (PMD), focusing on the interaction between the MDP regularizer (weighted by alpha) and the Drift regularizer (weighted by lambda). Using off-policy MDPO(h,D) instances on four small gymnax environments and over 500k training runs, the authors report L-shaped regions of well-performing temperature pairs, concluding that the two regularizers can partially substitute for each other but that their precise combination is critical for robustness. They also propose a robustness measure (Rbst_T), compare fixed versus annealed or learned temperature schedules, and report a linear relationship between the minimal required temperature and the environment's maximum return.

Significance. If the empirical claims hold, the paper fills a genuine gap: most PMD work is theoretical, and systematic empirical studies of the two regularization levels are scarce. The study is unusually extensive in its coverage of regularizer pairs (entropy, Tsallis, Lp, max; KL and Bregman divergences) and in its disclosure of the exact temperature grids and hyperparameters in Appendices C and D. The paper also offers a practical, falsifiable prediction (linear temperature scaling with reward magnitude) and a robustness ranking of regularizer pairs. These contributions could be useful for algorithm design and for guiding future empirical PMD research. The main limitations are the reliance on a hand-picked finite grid for the robustness measure, the lack of code/data release, and a few internal inconsistencies that currently prevent full reproducibility of the quantitative claims.

major comments (4)
  1. [Section 4.3 and Figure 4] There is an inconsistency in the definition of successful learning: the text states 'successful learning (normalized return ≥ 0.85)' while the Figure 4 caption states 'normalized return ≥ 0.75'. Since the plotted 'minimal required temperature' is exactly the threshold-crossing point, this discrepancy changes the fitted linear regressions and the reported slopes (0.0002 for alpha and 0.0045 for lambda). In addition, the 'sensible region' restriction, described only as 'sufficiently low λ and vice versa', is not defined by any precise rule and is applied post hoc, making the extraction procedure irreproducible as written. Please state the exact threshold, give an operational definition of the sensible region, and report sensitivity of the fitted lines to both choices.
  2. [Appendix B, Eq. (23) and Table 1] The robustness measure Rbst_T integrates the performance frequency over the fixed 29×29 temperature grid Φ of Appendix C. This grid is highly non-uniform: alpha is sampled every 0.001 below 0.1 and only every 0.1 above, while lambda is sampled quasi-logarithmically from 0 to 5×10^4. Because the measure counts grid points, the Table 1 ranking and the area-under-curve values in Figure 2 reflect the chosen grid geometry as much as algorithm behavior. The paper does not provide any sensitivity analysis (e.g., against a uniform grid, a differently bounded grid, or random grid samples) nor release code/data, so the reader cannot tell whether the L-shaped regions and the robustness ordering are artifacts of grid placement. Since robustness is a central claim of the abstract, this needs to be addressed.
  3. [Section 4.1 and Eq. (5)] For the baseline pair h=-H and D=D_KL, the PMD minimization in Eq. (5) has a closed-form solution of the form π_{k+1} ∝ π_k^{λ/(α+λ)} exp(Q/(α+λ)). The policy therefore depends on α and λ largely through the sum α+λ and the ratio λ/(α+λ). The L-shaped well-performing region in Figure 1a may thus be at least partly a mathematical consequence of this two-parameter dependence rather than an empirical discovery about substitutability of the two regularizers. The paper does not discuss this degeneracy. Please analyze the baseline in terms of the re-parameterized variables (α+λ and λ/(α+λ)), and show explicitly that the L-shape and the substitutability interpretation are not fully explained by this structure. For the non-entropic, non-KL pairs the degeneracy does not apply, but the baseline is the paper's main illustrative example and should be treated carefully.
  4. [Section 3 (Experiments), Figure 1 and Table 1] Each heatmap cell is computed from only N=5 training seeds per environment, and no standard errors or confidence intervals are reported for the heatmaps or for the robustness frequencies used in Table 1. With a total of 5 seeds per configuration per environment, the cell means are subject to substantial seed noise, and the threshold-crossing events that define robustness could be unstable. Since the central claims rest on the shape of the heatmaps and on the frequencies in Figure 2, please provide uncertainty quantification (e.g., bootstrap confidence intervals over seeds) or otherwise demonstrate that the L-shape and the robustness ordering are stable with respect to seed noise.
minor comments (4)
  1. [Figure 2 and Appendix C] The number of temperature configurations is inconsistent: Figure 2 says '784 temperature configurations', Section 4.2 says 'out of 784' in the text but 'out of 841' in the Figure 3 caption, and Appendix C states 29×29 = 841. Since the counts appear in the denominator of the frequency and robustness measures, please correct the inconsistent numbers and ensure the computation is uniform throughout.
  2. [Figure 3 caption] The caption describes 'the right part of Figure 3' and 'the left and right of each pair of columns' in a way that is confusing; the intended layout should be described more clearly, and the caption should specify which block corresponds to which algorithm and which percentile.
  3. [Section 4.1, Figure 2] The x-axis of Figure 2 is labeled 'Performance Threshold' and the y-axis 'Frequency', but the caption does not state whether the curves are averaged over environments and seeds in the same way as Eq. (9). Please clarify the exact definition of the performance values underlying the frequency curves.
  4. [Section 4.2, Figure 3] The claim that 'constant λ outperforms the linear annealing variant' is based on the top 1% and top 10% of configurations, which is an arbitrary percentile choice. Please report at least one aggregate measure (e.g., the full robustness metric) to show that the conclusion is not specific to the chosen percentiles.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's claims are empirical summaries of measured heat maps and self-defined robustness frequencies, not derivations that reduce to their inputs.

full rationale

No load-bearing circular step is present. The paper offers no first-principles derivation whose output equals its input; it is a large-scale empirical study. The central claims—partial substitutability of Drift and MDP regularization and sensitivity to their precise combination—are supported by measured heat maps (Figure 1) and by the self-defined robustness frequency Rbst_T in Eqs. (22)-(23). That metric is defined over the hand-picked 29x29 temperature grid in Appendix C, so the Table 1 ranking is grid-relative; however, the paper does not present Rbst_T as an external predictor or as a derived quantity, so this is a definitional scope limitation rather than a circular reduction. The only fitted quantities are the least-squares lines in Figure 4, explicitly labeled as regressions with R^2 values; they are post hoc summaries of the extracted minima, not predictions on held-out data, so the 'fitted input called prediction' pattern does not apply. The theoretical citations (Lan 2023, Zhan et al. 2023, Grudzien et al. 2022, Tomar et al. 2020) are external to the author set, and none is invoked as a uniqueness theorem or to forbid alternatives. Reproducibility concerns do exist—the success threshold for Figure 4 is inconsistent (text says >=0.85, caption says >=0.75), and the 'sensible region' restriction is post hoc—but these affect validity and reproducibility, not circularity. Accordingly, the score is 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claims rest on the chosen temperature grids, the representativeness of the small environments, and the correctness of the off-policy implementation, rather than on a mathematical derivation. The only fitted numbers are the linear regression coefficients in Figure 4 and the hand-picked performance threshold for extracting minimal temperatures.

free parameters (5)
  • slope of linear fit for alpha (Figure 4a) = 0.0002
    Fitted to the extracted minimal alpha values across rescaled CartPole environments to support the claimed linear scaling.
  • intercept of linear fit for alpha (Figure 4a) = -0.0171
    Fitted intercept for the alpha versus maximum return relationship.
  • slope of linear fit for lambda (Figure 4b) = 0.0045
    Fitted to the extracted minimal lambda values across rescaled CartPole environments to support the claimed linear scaling.
  • intercept of linear fit for lambda (Figure 4b) = 0.1433
    Fitted intercept for the lambda versus maximum return relationship.
  • performance threshold for 'successful learning' in Figure 4 = 0.85 in text, 0.75 in figure caption
    Hand-chosen threshold used to extract minimal temperatures; the inconsistency between text and caption is a red flag.
assumptions (4)
  • domain assumption The off-policy actor-critic implementation with the given hyperparameters is a faithful approximation of the Policy Mirror Descent update in Eq. (7).
    The paper acknowledges that theoretical guarantees do not translate to practice and uses a neural-network approximation; the results depend on this approximation being representative of PMD behavior.
  • domain assumption The discrete, small environments (CartPole, Acrobot, Catch, DeepSea) and the training horizon of 1e6 steps are sufficient to draw general conclusions about regularization in PMD.
    The paper itself notes in Section 6 that findings reflect performance within this training horizon and calls for extensions to larger environments.
  • standard math Standard convex analysis facts about Bregman divergences and the convexity of the regularizers (negative Shannon/Tsallis entropy, Lp norms, max function) are correct.
    Used in Appendix A to define the regularizers and their Bregman divergences.
  • domain assumption The maximum and minimum returns used for normalization are correctly computed per environment.
    Used in Appendix C to rescale returns; errors here would change all reported performance values.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On the Effect of Regularization in Policy Mirror Descent." pith.science (2026). https://pith.science/paper/QUJY3DGG

@misc{pith2026250708718,
  author       = {Pith},
  title        = {Pith review of: On the Effect of Regularization in Policy Mirror Descent},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QUJY3DGG}},
  note         = {Machine review of arXiv:2507.08718}
}
read the original abstract

Policy Mirror Descent (PMD) has emerged as a unifying framework in reinforcement learning (RL) by linking policy gradient methods with a first-order optimization method known as mirror descent. At its core, PMD incorporates two key regularization components: (i) a distance term that enforces a trust region for stable policy updates and (ii) an MDP regularizer that augments the reward function to promote structure and robustness. While PMD has been extensively studied in theory, empirical investigations remain scarce. This work provides a large-scale empirical analysis of the interplay between these two regularization techniques, running over 500k training seeds on small RL environments. Our results demonstrate that, although the two regularizers can partially substitute each other, their precise combination is critical for achieving robust performance. These findings highlight the potential for advancing research on more robust algorithms in RL, particularly with respect to hyperparameter sensitivity.

Figures

Figures reproduced from arXiv: 2507.08718 by the authors.

Figure 1
Figure 1. Mean normalized return after training of MDPO( [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Performance frequency curves for MDPO(h, D) instances, showing the propor￾tion of 784 temperature configurations reach￾ing each performance level. While high￾performance regions (≥ 0.98) narrow simi￾larly, regularizers h and D significantly af￾fect the frequency of achieving ≥ 90% per￾formance. Labels correspond to table 1 [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Mean & standard deviations of Top 1% and Top 10% performing hyperparameter config [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Minimal required temperature for successful learning (normalized return [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Different MDPO(h, D) configurations with constant temperatures 15 [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: MDPO(−H, DKL) with constant temperatures on different environments 0.0 0.0005 0.025 1.0 25.0 2500.0 1.0 0.1 0.01 0.005 0.0 0.4 0.6 0.8 1.0 Performance (a) h = −H0.5 & D = DKL 0.0 0.0005 0.025 1.0 25.0 2500.0 1.0 0.1 0.01 0.005 0.0 0.4 0.6 0.8 1.0 Performance (b) h = ||…
Figure 7
Figure 7. Figure 7: MDPO(h, D) for different h, D pairs 0.0 0.0005 0.025 1.0 25.0 2500.0 1.0 0.1 0.01 0.005 0.0 0.4 0.6 0.8 1.0 Performance (a) h = −H & D = B−H0.5 0.0 0.0005 0.025 1.0 25.0 2500.0 1.0 0.1 0.01 0.005 0.0 0.4 0.6 0.8 1.0 Performance (b) h = −H & D = B−H1.5 0.0 0.0005 0.025 …
Figure 8
Figure 8. Figure 8: MDPO(h, D) for different h, D pairs 0.0 0.0005 0.025 1.0 25.0 2500.0 1.0 0.1 0.01 0.005 0.0 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Performance (a) h = max & D = DKL 0.0 0.0005 0.025 1.0 25.0 2500.0 1.0 0.1 0.01 0.005 0.0 0.2 0.4 0.6 0.8 1.0 Performance (b) h = max & D = Bmax 0.0 …
Figure 9
Figure 9. Figure 9: MDPO(h, D) for different h, D pairs 16 [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: MDPO(−H, DKL) for different temperature scheduling schemes 0.0 0.0005 0.025 1.0 25.0 2500.0 1.0 0.1 0.01 0.005 0.0 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Performance (a) α = learned const., λ = const. 0.0 0.0005 0.025 1.0 25.0 2500.0 1.0 0.1 0.01 0.005 0.0 0.2 0.4 0.6 0.8 1.0 Pe…
Figure 11
Figure 11. Figure 11: MDPO(−H, DKL) for different temperature scheduling schemes 0.0 0.0005 0.025 1.0 25.0 2500.0 1.0 0.1 0.01 0.005 0.0 0.4 0.6 0.8 1.0 Performance (a) α = const., λ = const. 0.0 0.0005 0.025 1.0 25.0 2500.0 1.0 0.1 0.01 0.005 0.0 0.2 0.4 0.6 0.8 1.0 Performance (b) α = co…
Figure 12
Figure 12. Figure 12: MDPO(−H0.5, B−H0.5 ) for different temperature scheduling schemes 0.0 0.0005 0.025 1.0 25.0 2500.0 1.0 0.1 0.01 0.005 0.0 0.4 0.6 0.8 1.0 Performance (a) α = learned const., λ = const. 0.0 0.0005 0.025 1.0 25.0 2500.0 1.0 0.1 0.01 0.005 0.0 0.2 0.4 0.6 0.8 1.0 Perform…
Figure 13
Figure 13. Figure 13: MDPO(−H0.5, B−H0.5 ) for different temperature scheduling schemes 17 [PITH_FULL_IMAGE:figures/full_fig_p017_13.png]
Figure 14
Figure 14. Figure 14: MDPO(−H, DKL) with constant temperatures on CartPole with different maximum re￾turns 0.0 0.0005 0.025 1.0 25.0 2500.0 1.0 0.1 0.01 0.005 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Performance (a) Maximum Return = 333 0.0 0.0005 0.025 1.0 25.0 2500.0 1.0 0.1 0.01 0.005 0.0 0.1 0.2 0.…
Figure 15
Figure 15. Figure 15: MDPO(−H, DKL) with constant temperatures on CartPole with different maximum re￾turns 0.0 0.0005 0.025 1.0 25.0 2500.0 1.0 0.1 0.01 0.005 0.0 0.05 0.10 0.15 0.20 Performance (a) Maximum Return = 100 0.0 0.0005 0.025 1.0 25.0 2500.0 1.0 0.1 0.01 0.005 0.0 0.02 0.04 0.06…
Figure 16
Figure 16. Figure 16: MDPO(−H, DKL) with constant temperatures on CartPole with different maximum re￾turns 18 [PITH_FULL_IMAGE:figures/full_fig_p018_16.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Priced Motion Through Optimal Faces: A Normal-Fan Geometry for Non-Stationary Adversarial MDPs

    cs.LG 2026-06 unverdicted novelty 7.0 of 10

    Introduces priced face-crossing via normal-fan geometry on occupancy polytopes to decompose dynamic regret into intrinsic motion cost plus within-face error in non-stationary adversarial MDPs.

Reference graph

Works this paper leans on

43 extracted references · 21 canonical work pages · cited by 1 Pith paper

  1. [1]

    Reinforcement learning: Theory and algorithms

    Alekh Agarwal, Nan Jiang, Sham M Kakade, and Wen Sun. Reinforcement learning: Theory and algorithms. CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep, 32: 0 96, 2019

    2019

  2. [2]

    Deep Reinforcement Learning at the Edge of the Statistical Precipice

    Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep Reinforcement Learning at the Edge of the Statistical Precipice . In Advances in Neural Information Processing Systems , volume 34, pp.\ 29304--29320. Curran Associates, Inc., 2021

    work page 2021

  3. [3]

    A novel framework for policy mirror descent with general parameterization and linear convergence

    Carlo Alfano, Rui Yuan, and Patrick Rebeschini. A novel framework for policy mirror descent with general parameterization and linear convergence. Advances in Neural Information Processing Systems, 36: 0 30681--30725, 2023

    work page 2023

  4. [4]

    Learning mirror maps in policy mirror descent

    Carlo Alfano, Sebastian Rene Towers, Silvia Sapora, Chris Lu, and Patrick Rebeschini. Learning mirror maps in policy mirror descent. In Seventeenth European Workshop on Reinforcement Learning, 2024

    work page 2024

  5. [5]

    What matters in on-policy reinforcement learning? a large-scale empirical study

    Marcin Andrychowicz, Anton Raichuk, Piotr Sta \'n czyk, Manu Orsini, Sertan Girgin, Raphael Marinier, L \'e onard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, et al. What matters in on-policy reinforcement learning? a large-scale empirical study. arXiv preprint arXiv:2006.05990, 2020

    arXiv 2006

  6. [6]

    First-order methods in optimization

    Amir Beck. First-order methods in optimization. SIAM, 2017

    2017

  7. [7]

    Mirror descent and nonlinear projected subgradient methods for convex optimization

    Amir Beck and Marc Teboulle. Mirror descent and nonlinear projected subgradient methods for convex optimization. Operations Research Letters, 31 0 (3): 0 167--175, May 2003. ISSN 01676377. doi:10.1016/S0167-6377(02)00231-6

    doi:10.1016/s0167-6377(02)00231-6 2003

  8. [8]

    Openai gym

    Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. CoRR, abs/1606.01540, 2016. URL http://arxiv.org/abs/1606.01540

    arXiv 2016

Show all 43 references
  1. [9]

    Learning Pessimism for Reinforcement Learning

    Edoardo Cetin and Oya Celiktutan. Learning Pessimism for Reinforcement Learning . Proceedings of the AAAI Conference on Artificial Intelligence, 37 0 (6): 0 6971--6979, June 2023. ISSN 2374-3468, 2159-5399. doi:10.1609/aaai.v37i6.25852

    2023 doi

  2. [10]

    Path consistency learning in tsallis entropy regularized mdps

    Yinlam Chow, Ofir Nachum, and Mohammad Ghavamzadeh. Path consistency learning in tsallis entropy regularized mdps. In International conference on machine learning, pp.\ 979--988. PMLR, 2018

    2018

  3. [11]

    Implementation matters in deep policy gradients: A case study on ppo and trpo

    Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep policy gradients: A case study on ppo and trpo. arXiv preprint arXiv:2005.12729, 2020

    2005 arXiv

  4. [12]

    A Connection between One-Step RL and Critic Regularization in Reinforcement Learning

    Benjamin Eysenbach, Matthieu Geist, Sergey Levine, and Ruslan Salakhutdinov. A Connection between One-Step RL and Critic Regularization in Reinforcement Learning . In Proceedings of the 40th International Conference on Machine Learning , pp.\ 9485--9507. PMLR, July 2023

    2023

  5. [13]

    Addressing function approximation error in actor-critic methods

    Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pp.\ 1587--1596. PMLR, 2018

    2018

  6. [14]

    A Theory of Regularized Markov Decision Processes

    Matthieu Geist, Bruno Scherrer, and Olivier Pietquin. A Theory of Regularized Markov Decision Processes . In Proceedings of the 36th International Conference on Machine Learning , pp.\ 2160--2169. PMLR, May 2019

    2019

  7. [15]

    Mirror learning: A unifying framework of policy optimisation

    Jakub Grudzien, Christian A Schroeder De Witt, and Jakob Foerster. Mirror learning: A unifying framework of policy optimisation. In International Conference on Machine Learning, pp.\ 7825--7844. PMLR, 2022

    2022

  8. [16]

    Reinforcement Learning with Deep Energy-Based Policies

    Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement Learning with Deep Energy-Based Policies . In Proceedings of the 34th International Conference on Machine Learning , pp.\ 1352--1361. PMLR, July 2017

    2017

  9. [17]

    Soft Actor-Critic : Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor

    Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft Actor-Critic : Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor . In Proceedings of the 35th International Conference on Machine Learning , pp.\ 1861--1870. PMLR, July 2018 a

    2018

  10. [18]

    Soft actor-critic algorithms and applications

    Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905, 2018 b

    2018 arXiv

  11. [19]

    Policy mirror descent for reinforcement learning: Linear convergence, new sampling complexity, and generalized problem classes

    Guanghui Lan. Policy mirror descent for reinforcement learning: Linear convergence, new sampling complexity, and generalized problem classes. Mathematical Programming, 198 0 (1): 0 1059--1106, March 2023. ISSN 1436-4646. doi:10.1007/s10107-022-01816-5

    2023 doi

  12. [20]

    Primal-dual first-order methods with iteration-complexity for cone programming

    Guanghui Lan, Zhaosong Lu, and Renato DC Monteiro. Primal-dual first-order methods with iteration-complexity for cone programming. Mathematical Programming, 126 0 (1): 0 1--29, 2011

    2011

  13. [21]

    gymnax : A JAX -based reinforcement learning environment library, 2022

    Robert Tjarko Lange. gymnax : A JAX -based reinforcement learning environment library, 2022. URL http://github.com/RobertTLange/gymnax

    2022

  14. [22]

    Simba: Simplicity bias for scaling up parameters in deep reinforcement learning

    Hojoon Lee, Dongyoon Hwang, Donghu Kim, Hyunseung Kim, Jun Jet Tai, Kaushik Subramanian, Peter R Wurman, Jaegul Choo, Peter Stone, and Takuma Seno. Simba: Simplicity bias for scaling up parameters in deep reinforcement learning. arXiv preprint arXiv:2410.09754, 2024

    arXiv

  15. [23]

    Tsallis reinforcement learning: A unified framework for maximum entropy reinforcement learning

    Kyungjae Lee, Sungyub Kim, Sungbin Lim, Sungjoon Choi, and Songhwai Oh. Tsallis reinforcement learning: A unified framework for maximum entropy reinforcement learning. arXiv preprint arXiv:1902.00137, 2019

    1902 arXiv

  16. [24]

    Homotopic policy mirror descent: policy convergence, algorithmic regularization, and improved sample complexity

    Yan Li, Guanghui Lan, and Tuo Zhao. Homotopic policy mirror descent: policy convergence, algorithmic regularization, and improved sample complexity. Mathematical Programming, pp.\ 1--57, 2023

    2023

  17. [25]

    Neural trust region/proximal policy optimization attains globally optimal policy

    Boyi Liu, Qi Cai, Zhuoran Yang, and Zhaoran Wang. Neural trust region/proximal policy optimization attains globally optimal policy. Advances in neural information processing systems, 32, 2019

    2019

  18. [26]

    Discovered policy optimisation

    Chris Lu, Jakub Kuba, Alistair Letcher, Luke Metz, Christian Schroeder de Witt, and Jakob Foerster. Discovered policy optimisation. Advances in Neural Information Processing Systems, 35: 0 16455--16468, 2022

    2022

  19. [27]

    Algaedice: Policy gradient from arbitrary experience

    Ofir Nachum, Bo Dai, Ilya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans. Algaedice: Policy gradient from arbitrary experience. arXiv preprint arXiv:1912.02074, 2019

    1912 arXiv

  20. [28]

    Overestimation, overfitting, and plasticity in actor-critic: the bitter lesson of reinforcement learning

    Michal Nauman, Micha Bortkiewicz, Piotr Mi o \'s , Tomasz Trzci \'n ski, Mateusz Ostaszewski, and Marek Cygan. Overestimation, overfitting, and plasticity in actor-critic: the bitter lesson of reinforcement learning. arXiv preprint arXiv:2403.00514, 2024

    2024 arXiv

  21. [29]

    Bigger, regularized, optimistic: scaling for compute and sample efficient continuous control

    Michal Nauman, Mateusz Ostaszewski, Krzysztof Jankowski, Piotr Mi o \'s , and Marek Cygan. Bigger, regularized, optimistic: scaling for compute and sample efficient continuous control. Advances in Neural Information Processing Systems, 37: 0 113038--113071, 2025

    2025

  22. [30]

    Behaviour suite for reinforcement learning

    Ian Osband, Yotam Doron, Matteo Hessel, John Aslanides, Eren Sezener, Andre Saraiva, Katrina McKinney, Tor Lattimore, Csaba Szepesvari, Satinder Singh, et al. Behaviour suite for reinforcement learning. arXiv preprint arXiv:1908.03568, 2019

    1908 arXiv

  23. [31]

    On pathologies in kl-regularized reinforcement learning from expert demonstrations

    Tim GJ Rudner, Cong Lu, Michael A Osborne, Yarin Gal, and Yee Teh. On pathologies in kl-regularized reinforcement learning from expert demonstrations. Advances in Neural Information Processing Systems, 34: 0 28376--28389, 2021

    2021

  24. [32]

    Trust region policy optimization

    John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp.\ 1889--1897. PMLR, 2015

    2015

  25. [33]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  26. [34]

    Adaptive trust region policy optimization: Global convergence and faster rates for regularized mdps

    Lior Shani, Yonathan Efroni, and Shie Mannor. Adaptive trust region policy optimization: Global convergence and faster rates for regularized mdps. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp.\ 5668--5675, 2020

    2020

  27. [35]

    Reinforcement learning: an introduction

    Richard S Sutton. Reinforcement learning: an introduction. A Bradford Book, 2018

    2018

  28. [36]

    Regularized rl

    Daniil Tiapkin, Denis Belomestny, Daniele Calandriello, Eric Moulines, Alexey Naumov, Pierre Perrault, Michal Valko, and Pierre Menard. Regularized rl. arXiv preprint arXiv:2310.17303, 2023

    2023 arXiv

  29. [37]

    Mirror descent policy optimization

    Manan Tomar, Lior Shani, Yonathan Efroni, and Mohammad Ghavamzadeh. Mirror descent policy optimization. arXiv preprint arXiv:2005.09814, 2020

    2005 arXiv

  30. [38]

    A general class of surrogate functions for stable and efficient reinforcement learning

    Sharan Vaswani, Olivier Bachem, Simone Totaro, Robert M \"u ller, Shivam Garg, Matthieu Geist, Marlos C Machado, Pablo Samuel Castro, and Nicolas Le Roux. A general class of surrogate functions for stable and efficient reinforcement learning. In International Conference on Art...

    2022

  31. [39]

    Leverage the average: an analysis of kl regularization in rl

    Nino Vieillard, Tadashi Kozuno, Bruno Scherrer, Olivier Pietquin, R \'e mi Munos, and Matthieu Geist. Leverage the average: an analysis of kl regularization in rl. arXiv preprint arXiv:2003.14089, 2020

    2003 arXiv

  32. [40]

    On the convergence rates of policy gradient methods

    Lin Xiao. On the convergence rates of policy gradient methods. Journal of Machine Learning Research, 23 0 (282): 0 1--36, 2022

    2022

  33. [41]

    Policy mirror descent for regularized reinforcement learning: A generalized framework with linear convergence

    Wenhao Zhan, Shicong Cen, Baihe Huang, Yuxin Chen, Jason D Lee, and Yuejie Chi. Policy mirror descent for regularized reinforcement learning: A generalized framework with linear convergence. SIAM Journal on Optimization, 33 0 (2): 0 1061--1091, 2023

    2023

  34. [42]

    Modeling purposeful adaptive behavior with the principle of maximum causal entropy

    Brian D Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. Carnegie Mellon University, 2010

    2010

  35. [43]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.