Global Optimality for Constrained Exploration via Penalty Regularization

Florian Wolf; Ilyas Fatkhullin; Niao He

arxiv: 2604.28144 · v1 · submitted 2026-04-30 · 💻 cs.LG · math.OC

Global Optimality for Constrained Exploration via Penalty Regularization

Florian Wolf , Ilyas Fatkhullin , Niao He This is my paper

Pith reviewed 2026-05-07 05:19 UTC · model grok-4.3

classification 💻 cs.LG math.OC

keywords constrained explorationpolicy gradiententropy maximizationpenalty regularizationlast-iterate convergenceoccupancy measurereinforcement learninghidden convexity

0 comments

The pith

A quadratic-penalty policy gradient method achieves global last-iterate convergence for constrained maximum-entropy exploration in reinforcement learning.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces the Policy Gradient Penalty (PGP) method to maximize the entropy of the state-action occupancy measure subject to general convex constraints such as safety or resource limits. Earlier policy-gradient approaches delivered only weak regret bounds or guarantees on averaged iterates rather than a single deployable policy. PGP adds quadratic penalties to the constraints, builds pseudo-rewards that recover the gradient of the penalized objective through the standard Policy Gradient Theorem, and establishes the smoothness needed for convergence analysis. By invoking hidden convexity of the penalized problem together with strong duality, the method guarantees that the final policy reaches an ε-optimal constrained entropy value while keeping constraint violation bounded by ε, even though policy parameterization renders the original problem non-convex.

Core claim

The Policy Gradient Penalty (PGP) method enforces general convex occupancy-measure constraints through quadratic-penalty regularization in a single-loop policy-space optimization. It constructs pseudo-rewards that yield gradient estimates of the penalized objective via the classical Policy Gradient Theorem. The penalized objective is shown to be regular and smooth. Leveraging hidden convexity and strong duality, PGP attains global last-iterate convergence to an ε-optimal constrained entropy value with ε-bounded constraint violation, despite the non-convexity induced by policy parameterization.

What carries the argument

Quadratic-penalty regularization applied to convex occupancy-measure constraints, which produces an unconstrained penalized objective whose gradients are obtained via pseudo-rewards and the Policy Gradient Theorem, with global last-iterate optimality recovered through hidden convexity and strong duality.

If this is right

A single trained policy can be deployed directly rather than requiring ergodic averaging of multiple policies.
The same convergence guarantees cover arbitrary convex occupancy constraints including safety, resource, and imitation requirements.
The approach remains applicable under general differentiable policy parameterizations that would otherwise render the problem non-convex.
Scalable implementations become feasible for continuous-control tasks once the pseudo-reward construction is in place.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same penalty-plus-duality route may extend to other non-convex RL objectives whose occupancy-measure formulations regain hidden convexity after regularization.
Practitioners could adapt the penalty schedule dynamically during training to tighten feasibility without sacrificing the last-iterate guarantee.
The framework indicates that many constrained RL problems become globally tractable once they are lifted to the occupancy-measure space and then penalized.

Load-bearing premise

The occupancy-measure constraints are convex and the resulting penalized objective admits the smoothness and hidden-convexity properties required for the policy-gradient updates and the duality-based optimality transfer to be valid.

What would settle it

A grid-world or continuous-control run in which the final iterate of PGP produces a constrained entropy value more than ε worse than the known optimum or a constraint violation larger than ε, even after the penalty parameter is driven to the regime where theory predicts convergence.

Figures

Figures reproduced from arXiv: 2604.28144 by Florian Wolf, Ilyas Fatkhullin, Niao He.

**Figure 1.** Figure 1: Solving constrained maximumentropy exploration using Policy Gradient Penalty Method (PGP), unconstrained Maximum-Entropy (ME) exploration, and the primal-dual (PDPG) method of (Ying et al. 2025). Policies are visualized in view at source ↗

**Figure 3.** Figure 3: PointMass: State-Space Coverage. Relative visit frequencies of the SAC reference policy (trained to reach (⋆)) and PGP-policies evaluated on 30 start positions. Reference policy concentrates near the center, while increasing Rmax yields progressively more uniform state-space coverage. Reward Constraint. The SafeCartpole task (Dulac-Arnold et al. 2021) imposes a (linear) symmetric box constraint R(λ πθ ) :=… view at source ↗

**Figure 4.** Figure 4: (CME) on SafeCartpole. Trajectories illustrate diverse exploration of the full state space, consisting of the cart position and velocity as well as the pole angle and angular velocity view at source ↗

**Figure 5.** Figure 5: Overview of the grid-world with discrete state-action space (left) and the safe cartpole exploration view at source ↗

**Figure 6.** Figure 6: Ablation study of unconstrained entropy maximization on the environment Figure 5a with the (not considered) safety constraint (Rlin). We ablate different batch sizes B ∈ {8, 24, 72} and step sizes η ∈ {0.001, 0.01, 0.1}. Each curve shows the mean and one standard deviation, i.e. µ ± σ, for 10 different random seeds. E.1.1 Unconstrained Baseline For comparison, we also run the unconstrained baseline where w… view at source ↗

**Figure 7.** Figure 7: Ablation study of the entropy function value of our Algorithm 1 under linear constraints (Rlin) running on the environment Figure 5a. We ablate different batch sizes B ∈ {8, 24, 72}, penalty parameters β ∈ {5 · 10−4 , 0.001, 0.005, 0.01} and different step sizes η ∈ {0.001, 0.01, 0.1} of Stochastic Gradient Descent. Each curve shows the mean and one standard deviation of the entropy, i.e. µ ± σ, for 10 dif… view at source ↗

**Figure 8.** Figure 8: Ablation study of the (linear) constraint function value Equation (Rlin) for our Algorithm 1 running on the environment Figure 5a. We ablate different batch sizes B ∈ {8, 24, 72}, penalty parameters β ∈ {5 · 10−4 , 0.001, 0.005, 0.01} and different step sizes η ∈ {0.001, 0.01, 0.1} of Stochastic Gradient Descent. Each curve shows the mean and one standard deviation of the linear constraint function, i.e. µ… view at source ↗

**Figure 9.** Figure 9: Policies of Figure view at source ↗

**Figure 10.** Figure 10: Ablation study of the entropy function value for our Algorithm 1 under the relaxed (linear) constraint function value Equation (Rlin,relax) for our Algorithm 1 running on the environment Figure 5a. We ablate different batch sizes B ∈ {8, 24, 72}, larger penalty parameters β ∈ {0.013, 0.025, 0.125, 0.25} and different step sizes η ∈ {0.001, 0.01, 0.1} of Stochastic Gradient Descent. Each curve shows the … view at source ↗

**Figure 11.** Figure 11: Ablation study of the relaxed (linear) constraint function value Equation (Rlin,relax) for our Algorithm 1 running on the environment Figure 5a. We ablate different batch sizes B ∈ {8, 24, 72}, larger penalty parameters β ∈ {0.013, 0.025, 0.125, 0.25} and different step sizes η ∈ {0.001, 0.01, 0.1} of Stochastic Gradient Descent. Each curve shows the mean and one standard deviation of the linear constrain… view at source ↗

**Figure 12.** Figure 12: Final policies for maximum entropy with norm constraint (Rℓ2 ) for varying constraint shift parameter b. Each subfigure shows the learned final policy for different shifts b = 0.0, 0.2, 0.5, 1.0 (top-left to bottom-right) of the right-hand side of the constraint. Clearly, as b increases, the constraint becomes looser, allowing the learned policy to deviate further from the reference policy. b = 0.0 b = 0.… view at source ↗

**Figure 13.** Figure 13: Training progress for maximum entropy with norm constraint (Rℓ2 ) for varying constraint shift parameter b. Each subfigure shows the training curves for b = 0.0, 0.2, 0.5, 1.0 (top-left to bottomright). As b increases, the constraint becomes looser, allowing the learned policy to deviate further from the reference policy view at source ↗

**Figure 14.** Figure 14: PointMass: Control Trajectories. Comparison of actions sampled from πθ(· | s) for the reference SAC policy and PGP-trained policies with β = 10 and Rmax ∈ {2, 20, 200}, evaluated with the same fixed seed across 8 initial positions. Colored arrows indicate the actions selected by each policy at the current state. For PGP policies, the corresponding reference action πref(· | s) is overlaid in gray, illustra… view at source ↗

**Figure 15.** Figure 15: More fine-grained version of Figure 4b. Last iterate policy πθ (N) for constrained maximum entropy exploration on the SafeCartpole environment. The tracer’s color encodes the (normalized) pole’s angular velocity (dark blue for lower values and orange for higher values) which is part of the state space, i.e. has to be covered to maximize H. A more detailed visualization over several training iterations is … view at source ↗

**Figure 16.** Figure 16: Numbers in the legend indicate the iteration numbers used in Figure 4a and view at source ↗

**Figure 17.** Figure 17: Policy training evolution on the SafeCartpole environment for a fixed seed. Each column represents a training iteration (shown on the bottom), and each row represents a timestep in the trajectory (indicated at the top). The frames show how the learned policy evolves to successfully swing up the pendulum while respecting safety constraints, cf view at source ↗

read the original abstract

Efficient exploration is a central problem in reinforcement learning and is often formalized as maximizing the entropy of the state-action occupancy measure. While unconstrained maximum-entropy exploration is relatively well understood, real-world exploration is often constrained by safety, resource, or imitation requirements. This constrained setting is particularly challenging because entropy maximization lacks additive structure, rendering Bellman-equation-based methods inapplicable. Moreover, scalable approaches require policy parameterization, inducing non-convexity in both the objective and the constraints. To our knowledge, the only prior model-free policy-gradient approach for this setting under general policy parameterization is due to Ying et al. (2025). Unfortunately, their guarantees are limited to weak regret and ergodic averages, which do not imply that the final output is a single deployable policy that is near-optimal and nearly feasible. In this work we take a different approach to this problem, and propose Policy Gradient Penalty (PGP) method, a single-loop policy-space method that enforces general convex occupancy-measure constraints via quadratic-penalty regularization. PGP constructs pseudo-rewards that yield gradient estimates of the penalized objective, subsequently exploiting the classical Policy Gradient Theorem. We further establish the regularity of the penalized objective, providing the smoothness properties needed to justify the convergence of PGP. Leveraging hidden convexity and strong duality, we then establish global last-iterate convergence guarantees, attaining an $\epsilon$-optimal constrained entropy value with $\epsilon$ bounded constraint violation despite policy-induced non-convexity. We validate PGP through ablations on a grid-world benchmark and further demonstrate scalability on two challenging continuous-control tasks.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

PGP uses quadratic penalties on convex occupancy constraints to get global last-iterate convergence for max-entropy exploration, beating the ergodic guarantees in the only prior model-free method.

read the letter

The paper's central advance is the Policy Gradient Penalty (PGP) construction: a single-loop method that adds a quadratic penalty to the entropy objective, builds pseudo-rewards from the penalized problem, and runs ordinary policy gradients. They then prove the penalized objective is smooth and inherits hidden convexity, so strong duality transfers global optimality back to the policy parameters despite non-convex parameterization. This yields an epsilon-optimal constrained entropy value with bounded violation at the last iterate, which is stronger than the weak regret and average-case results in Ying et al. 2025. The approach is clean and directly addresses the lack of additive structure that blocks Bellman-style methods in the constrained setting. Experiments consist of grid-world ablations plus scalability checks on two continuous-control tasks, which at least show the method runs without obvious instability. The main soft spot is the transfer step: the claims rest on the occupancy constraints remaining convex after penalization and on the duality argument surviving the move from occupancy measures to arbitrary policy classes. If any regularity condition is missed in the proofs, the last-iterate guarantee does not hold. The empirical section is also thin on baselines and quantitative comparisons, so practical gains are not yet clear. This is for researchers working on theoretical constrained RL and safe exploration. A reader who cares about last-iterate guarantees in non-convex policy optimization will find the duality trick worth examining. The paper deserves peer review because the problem is real, the method is simple enough to implement, and the stated claims are specific enough to check against the full derivations.

Referee Report

1 major / 3 minor

Summary. The paper proposes the Policy Gradient Penalty (PGP) method for constrained maximum-entropy exploration in RL. It enforces general convex occupancy-measure constraints through quadratic-penalty regularization, derives pseudo-rewards to enable classical policy-gradient updates on the penalized objective, establishes smoothness and regularity of that objective, and invokes hidden convexity together with strong duality to prove global last-iterate convergence to an ε-optimal constrained entropy value with bounded constraint violation, even under non-convex policy parameterization. Empirical support is given via grid-world ablations and two continuous-control tasks.

Significance. If the hidden-convexity and strong-duality arguments are correct, the work supplies the first last-iterate global-optimality guarantee for this constrained setting under general policy parameterization, a clear improvement over the weak-regret/ergodic-average results of Ying et al. (2025). The single-loop penalty approach is practically attractive for safe or resource-constrained exploration. The technical device of transferring optimality via hidden convexity may have wider applicability to other non-convex occupancy-measure problems in RL.

major comments (1)

Abstract and convergence analysis: the central claim that strong duality transfers optimality from the convex occupancy-measure problem to the non-convex policy parameterization rests on the penalized objective admitting hidden convexity and satisfying a constraint qualification. The manuscript must explicitly state the qualification used (e.g., Slater’s condition or an interior-point assumption on the occupancy constraints) and verify that it remains valid after the quadratic penalty is introduced and the policy parameterization is applied.

minor comments (3)

Abstract: the citation “Ying et al. (2025)” should be expanded to a full bibliographic entry in the reference list and cross-checked for consistency with the in-text citation.
Experiments section: the precise convex occupancy-measure constraints used in the grid-world ablations and the two continuous-control tasks, together with the numerical values of the penalty coefficient, should be stated explicitly to support reproducibility.
Notation and algorithm description: the construction of the pseudo-rewards from the quadratic penalty term would benefit from an explicit equation label so that subsequent convergence arguments can refer to it directly.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the positive assessment of our work and for identifying this important clarification point regarding the constraint qualification. We address the comment below and will revise the manuscript accordingly.

read point-by-point responses

Referee: Abstract and convergence analysis: the central claim that strong duality transfers optimality from the convex occupancy-measure problem to the non-convex policy parameterization rests on the penalized objective admitting hidden convexity and satisfying a constraint qualification. The manuscript must explicitly state the qualification used (e.g., Slater’s condition or an interior-point assumption on the occupancy constraints) and verify that it remains valid after the quadratic penalty is introduced and the policy parameterization is applied.

Authors: We agree that the constraint qualification should be stated explicitly rather than left implicit. In the revised manuscript we will add a dedicated paragraph in the convergence analysis (Section 4) stating that we invoke Slater’s condition on the original convex occupancy-measure problem: there exists an occupancy measure μ* satisfying g(μ*) < 0 for all inequality constraints (with a positive margin). We will then verify that this condition is preserved after quadratic-penalty regularization: the penalty term is identically zero on the feasible set and strictly positive outside it, so any strictly feasible μ* remains strictly feasible for the penalized objective; moreover, because the penalty is convex in the occupancy measure, the hidden-convexity property of the penalized problem is unchanged. For the non-convex policy parameterization, hidden convexity already guarantees that the policy-induced occupancy measures can attain the same optimum as the convex relaxation; the Slater condition is a property of the occupancy-measure feasible set and does not need to be re-verified for each parameterization (provided the policy class is sufficiently expressive, which is assumed throughout and validated empirically). We will also insert a brief reference to this assumption in the abstract. These changes will be made without altering any theorems or proofs. revision: yes

Circularity Check

0 steps flagged

No significant circularity; derivation relies on classical theorems

full rationale

The paper's derivation chain begins from the standard Policy Gradient Theorem applied to a quadratic-penalty regularized objective, then invokes established smoothness properties, hidden convexity of the penalized entropy, and strong duality to transfer global optimality from the convex occupancy-measure space back to the non-convex policy parameterization. These steps cite the classical Policy Gradient Theorem and standard convex-optimization duality results rather than any self-defined quantity, fitted parameter, or prior result by the same authors. The single external citation to Ying et al. (2025) is to independent prior work on the same problem and is not used to justify uniqueness or to close the main argument. No equation reduces to its own input by construction, and the central last-iterate guarantee remains independently supported by the cited external theorems.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The central claim rests on two domain assumptions about the penalized formulation that are not standard in unconstrained RL and are not independently verified in the abstract.

axioms (2)

domain assumption The penalized objective is smooth and possesses hidden convexity
Invoked to justify both the applicability of policy-gradient methods and the global optimality transfer via duality.
domain assumption Strong duality holds for the constrained entropy problem after penalty regularization
Used to convert near-optimality of the penalized problem into near-feasibility and near-optimality of the original constrained problem.

pith-pipeline@v0.9.0 · 5583 in / 1340 out tokens · 75201 ms · 2026-05-07T05:19:51.542015+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

8 extracted references · 8 canonical work pages

[1]

PMLR, July 2023. A. Barakat, S. Chakraborty, P. Yu, P. Tokekar, and A. S. Bedi. Towards Scalable General Utility Reinforcement Learning: Occupancy Approximation, Sample Complexity and Global Optimality, Feb. 2025. D. P. Bertsekas.Nonlinear Programming. Athena Scientific, 2 edition, Sept. 1999. ISBN 978-1-886529-00-7. D. P. Bertsekas, editor.Convex Optimiz...

work page doi:10.1007/s10107-024-02057-4 2023
[2]

PMLR, Nov. 2020. B. Marin Moreno, M. Brégère, P. Gaillard, and N. Oudjane. MetaCURL: Non-stationary Concave Utility Rein- forcement Learning.Advances in Neural Information Processing Systems, 37:123091–123126, Dec. 2024. doi: 10.52202/079017-3912. B. Martinet. Brève communication. Régularisation d’inéquations variationnelles par approximations successives...

work page doi:10.52202/079017-3912 2020
[3]

ISSN 0037-9484. B. M. Moreno, M. Brégère, P. Gaillard, and N. Oudjane. Efficient Model-Based Concave Utility Reinforcement Learning through Greedy Mirror Descent. In S. Dasgupta, S. Mandt, and Y. Li, editors,International Conference on Artificial Intelligence and Statistics, 2-4 May 2024, Palau de Congressos, Valencia, Spain, volume 238 ofProceedings of M...

work page doi:10.1609/aaai.v34i04.5968 2024
[4]

PMLR, 2025. A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmai- son, A. Köpf, E. Z. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In H. M. Wallach, H. Larochelle...

work page doi:10.1007/s11590-016-1024-6 2025
[5]

doi: 10.1287/moor.8.2.231

ISSN 0364-765X, 1526-5471. doi: 10.1287/moor.8.2.231. A. Wachi, W. Hashimoto, X. Shen, and K. Hashimoto. Safe Exploration in Reinforcement Learning: A Generalized Formulation and Algorithms. InThirty-Seventh Conference on Neural Information Processing Systems, Nov. 2023. I.-J. Wang and J. C. Spall. Stochastic optimisation with inequality constraints using...

work page doi:10.1287/moor.8.2.231 2023
[6]

The domainΛ=c(Θ)is convex, the functionsH 1, H2 :Λ→Rare convex, i.e. satisfy fori= 1,2and for allu, v∈Λ and anyα∈[0,1] Hi((1−α)u+αv)≤(1−α)H i(u) +αH i(v)− (1−α)αµ H 2 ∥u−v∥ 2.(HC-1) Additionally, we assume (HC) admits a solutionu ∗ ∈Λwith its corresponding objective function valueF ∗ := H1(u∗) =F 1(c−1(u∗)). Throughout this appendix,F∗ denotes the minimum...

work page
[7]

IPPPM” refers to the inexact proximal point penalty framework, “GD

The mapc:Θ→Λis invertible and there exists aµ c >0such that for allθ, ˜θ∈Θit holds ∥c(θ)−c( ˜θ)∥ ≥µ c∥θ− ˜θ∥.(HC-2) Ifµ H >0, we refer to (HC) as(µ c, µH )-strongly convex. Note that in this more generic setting, we haveµc =µ λ of Assumption 2. We can map (CME) to (HC) viaH1 =−HandH 2 =R. Note that the condition (HC-2) along with Assumption 8 imply that t...

work page 2020
[8]

VF” refers to the value function, i.e. the critic we use in the REINFORCE estimate, “Lr

which is an extension of the Google DeepMind Con- trol suite (Todorov et al. 2012, Tassa et al. 2018). The pen- dulum starts in the resting downwards position and the goal is to safely explore the dynamics of the system with- out exceeding an imposed slider safety limits on the left and right. Fig. 5: Overview of the grid-world with discrete state-action ...

work page 2012

[1] [1]

PMLR, July 2023. A. Barakat, S. Chakraborty, P. Yu, P. Tokekar, and A. S. Bedi. Towards Scalable General Utility Reinforcement Learning: Occupancy Approximation, Sample Complexity and Global Optimality, Feb. 2025. D. P. Bertsekas.Nonlinear Programming. Athena Scientific, 2 edition, Sept. 1999. ISBN 978-1-886529-00-7. D. P. Bertsekas, editor.Convex Optimiz...

work page doi:10.1007/s10107-024-02057-4 2023

[2] [2]

PMLR, Nov. 2020. B. Marin Moreno, M. Brégère, P. Gaillard, and N. Oudjane. MetaCURL: Non-stationary Concave Utility Rein- forcement Learning.Advances in Neural Information Processing Systems, 37:123091–123126, Dec. 2024. doi: 10.52202/079017-3912. B. Martinet. Brève communication. Régularisation d’inéquations variationnelles par approximations successives...

work page doi:10.52202/079017-3912 2020

[3] [3]

ISSN 0037-9484. B. M. Moreno, M. Brégère, P. Gaillard, and N. Oudjane. Efficient Model-Based Concave Utility Reinforcement Learning through Greedy Mirror Descent. In S. Dasgupta, S. Mandt, and Y. Li, editors,International Conference on Artificial Intelligence and Statistics, 2-4 May 2024, Palau de Congressos, Valencia, Spain, volume 238 ofProceedings of M...

work page doi:10.1609/aaai.v34i04.5968 2024

[4] [4]

PMLR, 2025. A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmai- son, A. Köpf, E. Z. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In H. M. Wallach, H. Larochelle...

work page doi:10.1007/s11590-016-1024-6 2025

[5] [5]

doi: 10.1287/moor.8.2.231

ISSN 0364-765X, 1526-5471. doi: 10.1287/moor.8.2.231. A. Wachi, W. Hashimoto, X. Shen, and K. Hashimoto. Safe Exploration in Reinforcement Learning: A Generalized Formulation and Algorithms. InThirty-Seventh Conference on Neural Information Processing Systems, Nov. 2023. I.-J. Wang and J. C. Spall. Stochastic optimisation with inequality constraints using...

work page doi:10.1287/moor.8.2.231 2023

[6] [6]

The domainΛ=c(Θ)is convex, the functionsH 1, H2 :Λ→Rare convex, i.e. satisfy fori= 1,2and for allu, v∈Λ and anyα∈[0,1] Hi((1−α)u+αv)≤(1−α)H i(u) +αH i(v)− (1−α)αµ H 2 ∥u−v∥ 2.(HC-1) Additionally, we assume (HC) admits a solutionu ∗ ∈Λwith its corresponding objective function valueF ∗ := H1(u∗) =F 1(c−1(u∗)). Throughout this appendix,F∗ denotes the minimum...

work page

[7] [7]

IPPPM” refers to the inexact proximal point penalty framework, “GD

The mapc:Θ→Λis invertible and there exists aµ c >0such that for allθ, ˜θ∈Θit holds ∥c(θ)−c( ˜θ)∥ ≥µ c∥θ− ˜θ∥.(HC-2) Ifµ H >0, we refer to (HC) as(µ c, µH )-strongly convex. Note that in this more generic setting, we haveµc =µ λ of Assumption 2. We can map (CME) to (HC) viaH1 =−HandH 2 =R. Note that the condition (HC-2) along with Assumption 8 imply that t...

work page 2020

[8] [8]

VF” refers to the value function, i.e. the critic we use in the REINFORCE estimate, “Lr

which is an extension of the Google DeepMind Con- trol suite (Todorov et al. 2012, Tassa et al. 2018). The pen- dulum starts in the resting downwards position and the goal is to safely explore the dynamics of the system with- out exceeding an imposed slider safety limits on the left and right. Fig. 5: Overview of the grid-world with discrete state-action ...

work page 2012