Pith. sign in

REVIEW 3 major objections 6 minor 31 references

Safe Planning and Policy Optimization via World Model Learning

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SPOWL claims a model-based RL agent can jointly optimize task reward and safety by dynamically switching between short-horizon world-model planning and a Lagrangian safe policy, with safety thresholds that adapt as the agent improves.

desk verdict SPOWL's genuinely new adaptive threshold and switching rule are worth a serious look, but the near-zero violation claim rests on an unvalidated unbiasedness assumption about the cost-value ensemble on planner-explored actions. read the letter →

arxiv 2506.04828 v1 pith:BQSDWS6U submitted 2025-06-05 cs.AI

Artem Latyshev , Gregory Gorbov , Aleksandr I. Panov This is my paper
classification cs.AI
keywords safereinforcementlearningmodel-basedworldmodelsconstrainedMarkovdecisionprocessmodelpredictivecontrolLagrangianmethodsadaptivesafetythresholdscontinuous
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that a single model-based reinforcement learning agent can chase two goals at once: maximize task reward and keep safety violations near zero. It proposes SPOWL, which alternates between short-horizon planning inside a learned world model and executing actions from a safe policy trained with an augmented Lagrangian objective. The safety threshold used during planning is not fixed; it is re-computed from the current safe policy's own reward and cost estimates, so the planner is asked to improve on the policy rather than meet an externally chosen level. The authors test SPOWL on a standard safe continuous-control benchmark with Point, Car, Doggo, and Ant embodiments and report that it keeps episode costs close to zero while achieving higher returns than existing safe RL baselines. If those results hold, model-based safe RL would not need careful threshold tuning and could be applied to tasks where a single violation is costly.

What carries the argument

The load-bearing object is the value-equivalent implicit world model paired with a safe-improvement planning step. The model uses SimNorm latent states and ensemble MLPs for reward and cost value functions, with discrete regression via cross-entropy. In planning, each imagined trajectory is scored by $J^M = \sum_i \gamma^i \hat{r}_{t+i} + \gamma^H \hat{Q}^{\mathrm{avg}}$ and $J^M_c = \sum_i \gamma_c^i \hat{c}_{t+i} + \gamma_c^H \hat{Q}^{c,\mathrm{avg}}$. The adaptive thresholds $d_R$ and $d_c$ are the mean $J^M$ and $J^M_c$ over policy-prior action sequences; only trajectories with $J^M \ge d_R$ and $J^M_c \le d_c$ become elites. The decision module then uses ensemble averages $\hat{Q}^{\mathrm{avg}}(z_t, a_{\mathrm{plan}})$ and $\hat{Q}^{c,\mathrm{avg}}(z_t, a_{\mathrm{plan}})$ versus the policy's action to decide whether to commit to the plan. This machinery is what converts 'learn a world model' into 'improve over the safe policy in both objectives'.

What would settle it

Run SPOWL on a sparse-cost continuous-control task and compare violation rates when the cost ensemble is used as-is versus when the same ensemble predictions are artificially shifted downward before planning and switching. If violations stay near zero under the downward shift, the safety claim does not depend on cost-estimate accuracy; if violations rise, the near-zero safety guarantee is conditional on unbiased cost estimates. An oracle version with true costs plugged in would isolate the cost-model contribution.

Watch

Extended reading notes

Core claim

SPOWL's central claim is that a value-equivalent, implicit world model can drive both planning and policy optimization for safe control. The world model predicts latent states, rewards, costs, and ensembles of reward and cost value functions, without decoding observations. Planning is done by sampling short action sequences, scoring them with model estimates $J^M$ and $J^M_c$, and keeping the top sequences that beat the average reward and average cost of the safe policy's own action sequences. The final decision compares the ensemble-averaged value estimates: the plan is executed only if it is at least as good as the policy action in both reward and cost. The paper reports that this adaptive scheme achieves near-zero cost rates on Point, Car, Doggo, and Ant tasks, and that it outperforms fixed-threshold constrained cross-entropy planning and model-free safe RL baselines in both reward and safety.

Load-bearing premise

The near-zero violation results rest on the learned cost value estimates being accurate; if the world model underestimates costs, the planner can pick actions that are actually more costly than the safe policy, and the 'improvement' guarantee breaks down.

Editorial extensions

If this is right

  • If the framework is correct, safe model-based RL can operate without manually chosen planning thresholds; the policy's own estimates set the bar.
  • Near-zero evaluation-time violations can coexist with strong task performance, rather than being traded off against it.
  • Planning horizons can stay short (H=3), because long-term value is supplied by learned value functions, limiting per-step computation.
  • The framework handles high-dimensional continuous embodiments (Ant, Doggo) where prior model-based safe methods stagnate.
  • The same adaptive threshold idea transfers to other planners, since it only requires reward and cost value estimates of policy-prior actions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the adaptive threshold is anchored to the safe policy's own cost estimates, so an over-optimistic policy lowers the bar for the planner; the method therefore inherits rather than removes the policy's cost-estimation risk during early training.
  • Beyond the paper: the switching rule requires the plan to dominate the policy action in both reward and cost, so it will reject a plan that is much safer but slightly less rewarding; a weighted safety-reward criterion could cover a wider set of useful plans.
  • Beyond the paper: since the paper notes slow cost learning on PointGoal1 from data imbalance, a natural testable variant would reweight cost-critical transitions in the replay buffer to speed up cost value learning and see whether violation rates fall further.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes SPOWL, a model-based safe reinforcement learning framework built on a TD-MPC2-style implicit world model with latent cost and reward value ensembles, an augmented-Lagrangian-trained safe policy, and MPC planning with adaptive safety thresholds. The planner generates candidate action sequences from a policy prior and Gaussian sampling, filters them using thresholds derived from the policy prior's value estimates, and then switches to the plan only if its estimated reward and cost beat those of the policy under the same learned value functions. Experiments on SafetyGymnasium report comparisons against SafeDreamer, PPO-Lagrangian, CPO, and CVPO, together with ablations of thresholds, switching, ensemble aggregation, and decoder presence.

Significance. If the safety claims held, SPOWL would offer a practically valuable Pareto improvement over model-free safe RL baselines and over SafeDreamer, while also addressing objective mismatch through an implicit world model. The paper provides a concrete algorithmic recipe (Algorithms 1 and 2), a sensible design space exploration, and an ablation study of several components. The main strength is the adaptive-threshold idea, which is cheap and plausible. However, the central safety-improvement claim is currently definitional with respect to the learned cost estimator rather than validated against true costs, the statistical support is incomplete (no seed counts, no error bars on ablations), and Table 1 shows SafeDreamer achieving lower costs than SPOWL on several tasks. The contribution is promising but needs substantial revision before the claims can be accepted.

major comments (3)
  1. [Sections 4.3–4.4, Eq. (11)–(12), Algorithm 2] The safety-improvement claim is circular with respect to the learned cost estimator. The threshold dc is the mean of J_c^M over policy-prior trajectories, elite selection requires J_c^M(a) ≤ dc, and the final switch requires Q_c,avg(z_t, a_plan) ≤ Q_c,avg(z_t, π(z_t)). Thus any plan passing all tests is, by construction, safer than π according to the same estimator that defined the tests. If Q_c is biased low on actions underrepresented in the replay buffer, a plan can have higher true cost than π and still pass every filter. The paper's own Section 5.2 notes that cost components learn slowly on PointGoal1 due to data imbalance, and Section 5.3 shows that the ensemble aggregation choice materially changes violation rates. No uncertainty penalty, pessimism correction, or true-cost validation is applied at decision time. Please provide evidence that Q_c is calibrated for planner-explored actions, for example by comparing true costs of accepted versus rejected plans, or by introducing an explicit pessimism term.
  2. [Table 1] The claims of 'near-zero safety violations' and 'outperforming existing methods' need qualification. SafeDreamer achieves zero episode cost on PG2, PB1, CG1, DG1, and AG1, while SPOWL reports C = 4.8, 5.4, 1.0, 0.3, and 2.0 on those tasks, respectively. Because SafeDreamer's rewards are much lower, the comparison is a Pareto tradeoff rather than a dominance result. The text should state this explicitly and define what 'near-zero' means relative to episode length and cost threshold, so the reader can judge whether SPOWL's nonzero costs are acceptable.
  3. [Section 5 and Figures 3–9] The statistical evidence is incomplete. The number of seeds is not stated anywhere, and the ablation figures (Figures 3, 4, 6, 7, 8, 9) do not report error bars or significance tests. Table 1 reports ± values but without a seed count; some differences, such as cost rates on the order of 10^4 (e.g., 133 versus 160 for PG1), may be within noise. Without this information, the claims of 'significant improvements' and 'robust performance' are not fully supported. Please specify the number of seeds, report confidence intervals on all curves, and include significance tests or effect sizes for the main comparisons.
minor comments (6)
  1. [Eq. (4)] The threshold dM in the constrained MPC formulation is never defined; please define it or note that it is a generic placeholder.
  2. [Section 4.4 / Algorithm 2] The notation Qavg is ambiguous: Section 4.1 states that reward values use the minimum of two subsampled ensemble heads while cost values use the ensemble average, but the decision rule and Algorithm 2 use Qavg for both. Please clarify which aggregation is used in each comparison.
  3. [References] The reference list contains duplicates: [17] and [18] are the same OmniSafe paper, and [20] and [21] are both 'Constrained model-based reinforcement learning with robust cross-entropy method'. Please consolidate.
  4. [Eq. (10)] Equation (10) is hard to parse: the line 'Ψ, λk+1 l = ...' mixes the penalty term and the multiplier update. Please separate the definition of Ψ from the update rule and use consistent notation for the Augmented Lagrangian terms.
  5. [Algorithm 2] The initialization and shifting of the action sequence is confusing: 'a0 ← 0' and 'at+1 = (aN t+1, ..., aN t+H−1, 0) ← µN' are not clearly specified. Please spell out how the previous plan's tail is reused and how the first action is extracted.
  6. [Figure 10 caption] The caption says 'PG1-PointGoal1, PG1-CarGoal1'; the second abbreviation should presumably be CG1-CarGoal1.

Circularity Check

1 steps flagged · score 6.0 of 10

SPOWL's 'surpass safe policy' claim is defined by the same world-model estimates that set thresholds and authorize switching; the near-zero violation result is externally measured and independent, but the internal improvement claim is tautological.

  1. self definitional [Section 4.3, Eq. (12) and Section 4.4, Algorithm 2]
    "dR = 1 |{ˆat:t+H }| P a∈{ˆat:t+H } J M (a), dc = 1 |{ˆat:t+H }| P a∈{ˆat:t+H } J M c (a), (12) with selection criteria J M ≥ dR and J M c ≤ dc ... If ˆQavg(zt, aplan) ≥ ˆQavg(zt, π(zt)) and ˆQc,avg(zt, aplan) ≤ ˆQc,avg(zt, π(zt)), then aenv = aplan."

    The thresholds d_R and d_c are defined as the means of the world model's own estimates over the policy-prior trajectories. A trajectory is an 'improvement' exactly when its model-estimated reward is at least the policy-prior mean and its model-estimated cost is at most the policy-prior mean. The final switch applies the same learned estimators to the candidate plan and the policy action. Hence the claim that planning 'consistently selects actions that surpass safe policy suggestions in both performance and safety' (Abstract) is true by construction on the learned estimates, not by any independent validation.

full rationale

SPOWL's headline empirical results—near-zero cost violations and task reward in SafetyGymnasium—are evaluated with external environment metrics and are not circular. However, a central stated feature of the method is that it 'consistently selects actions that surpass safe policy suggestions in both performance and safety'. This claim is established by construction: in Section 4.3 the selection thresholds d_R and d_c are the means of the world model's J^M and J^M_c over the policy prior, and the elite criterion is J^M >= d_R and J^M_c <= d_c. In Section 4.4, the plan is executed only if Q_avg(a_plan) >= Q_avg(pi) and Q_c,avg(a_plan) <= Q_c,avg(pi), using the same learned value ensembles. Any plan that passes therefore beats the policy on the model's own estimates by definition. No uncertainty penalty, pessimism, or true-cost validation is applied at decision time. The paper acknowledges the cost model learns slowly on PointGoal1 and that ensemble aggregation choices change violation counts, so the 'surpass' property is not guaranteed on true costs. The external benchmarks support the overall method, but this specific internal 'surpass' claim is tautological. Score 6 reflects that one central prediction reduces by construction while the main empirical comparison remains independent.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim depends on a handful of hand-chosen hyperparameters and on the unverified accuracy of the cost value estimates used both for planning thresholds and for switching decisions. No new physical entities are introduced.

free parameters (5)
  • Safety threshold b = 0.1
    Sets the per-step cost value limit in the augmented Lagrangian policy loss (Section 4.2). Chosen by hand, it directly controls the exploration-safety trade-off.
  • Planning horizon H = 3
    Short horizon chosen to keep planning cheap and to limit model error accumulation (Section 5.2). It may miss longer-term costs.
  • Ensemble size x subsample = 5x5
    Selected from ablation in Section 5.3; other settings such as 2x5 or 5x2 change safety and reward, so this choice matters.
  • Penalty growth rate nu = not reported
    Controls the growth rate of the augmented Lagrangian penalty (Section 4.2); the paper does not report its value.
  • CEM planning hyperparameters = not reported
    Population size, number of planning iterations, elite count k, and sigma_init are not specified, making precise reproduction difficult.
assumptions (4)
  • domain assumption The world model's latent state z_t is value-equivalent, preserving all information needed to predict rewards and costs.
    Inherited from TD-MPC2 [12]; the paper does not verify this for cost prediction, and Section 5.2 notes cost learning is slow due to data imbalance.
  • domain assumption The ensemble-averaged cost value estimates are accurate enough for comparing planner actions against policy actions and for setting planning thresholds.
    The adaptive threshold and switching mechanism (Sections 4.3-4.4) rely on Qc estimates; Section 5.3 shows that changing the aggregation method to min or max significantly changes safety, so this is load-bearing.
  • ad hoc to paper The safe policy trained via augmented Lagrangian is a reliable action prior at all training stages, so thresholds derived from its predictions are sensible.
    The conclusion admits early training phases may have safety violations before optimal policies emerge, so the policy prior early in training is not necessarily safe.
  • standard math Standard stochastic gradient training and TD learning converge to useful approximations of the true Q and Qc functions on the collected replay data.
    Standard deep RL assumptions; not proven for sparse cost signals, which the paper acknowledges.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Safe Planning and Policy Optimization via World Model Learning." pith.science (2026). https://pith.science/paper/BQSDWS6U

@misc{pith2026250604828,
  author       = {Pith},
  title        = {Pith review of: Safe Planning and Policy Optimization via World Model Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BQSDWS6U}},
  note         = {Machine review of arXiv:2506.04828}
}
read the original abstract

Reinforcement Learning (RL) applications in real-world scenarios must prioritize safety and reliability, which impose strict constraints on agent behavior. Model-based RL leverages predictive world models for action planning and policy optimization, but inherent model inaccuracies can lead to catastrophic failures in safety-critical settings. We propose a novel model-based RL framework that jointly optimizes task performance and safety. To address world model errors, our method incorporates an adaptive mechanism that dynamically switches between model-based planning and direct policy execution. We resolve the objective mismatch problem of traditional model-based approaches using an implicit world model. Furthermore, our framework employs dynamic safety thresholds that adapt to the agent's evolving capabilities, consistently selecting actions that surpass safe policy suggestions in both performance and safety. Experiments demonstrate significant improvements over non-adaptive methods, showing that our approach optimizes safety and performance simultaneously rather than merely meeting minimum safety requirements. The proposed framework achieves robust performance on diverse safety-critical continuous control tasks, outperforming existing methods.

Figures

Figures reproduced from arXiv: 2506.04828 by the authors.

Figure 1
Figure 1. World model architecture. Neural networks implement all components, with the latent state space following the SimNorm scheme [12]. Cost and reward components use discrete regression optimization [12]. Dashed components generate targets for latent states to provide consistency. We employ discrete regression in log-transformed space to opti￾mize cost, reward, and value functions by minimizing cross-entropy. This appro… view at source ↗
Figure 2
Figure 2. SPOWL Planning Process. The agent evaluates imagined trajectories from: policy-generated sequences (Policy Prior), and samples from a normal distribution parameterized by statistics from previous iterations (Action Samples). Selection thresholds d R and d c are derived from the Policy Prior set. Selection Phase. We select improvement trajectories Aimpr using thresholds computed from the Policy Prior: d R = 1 |{aˆt:t… view at source ↗
Figure 3
Figure 3. Global estimation for ablation safety thresholds (dplan = 1, 3, 5, 10, 25) of Constrained Cross Entropy planning. SPOWL denotes agent with all components, “adaptive” uses only Adaptive Threshold without Dynamic Policy Switching [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Local estimation for ablation safety thresholds (dplan = 0.01, 0.1, 1) of Constrained Cross Entropy planning. SPOWL denotes agent with all components, “adaptive” uses only Adaptive Threshold without Dynamic Policy Switching. Our experiments suggest that adaptive thresh…
Figure 5
Figure 5. Figure 5: The performance of the Constrained Cross Entropy with global estimation starts to rise then the estimation falls below the threshold (dashed lines). The mean cost value estimation is high for early steps. 5.2 Switching Strategy We have discussed how using a world model…
Figure 7
Figure 7. Figure 7: Comparison of aggregation methods for target cost estimation. Average avoids both overestimation and underestimation extremes. For the Augmented Lagrangian policy loss (∆ in Equation (10)), we analyze subsampling effects across ensemble sizes, subsample counts, and agg…
Figure 9
Figure 9. Figure 9: Decoder ablation study with loss weights 0.01, 0.1, and 1. "No consistency" indicates weight=0 for consistency loss while keeping decoder weight at 20 (usual for consistency loss). performance while maintaining near-zero cost violations. 6 Conclusion This paper present…
Figure 10
Figure 10. Figure 10 [PITH_FULL_IMAGE:figures/full_fig_p009_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

31 extracted references · 25 canonical work pages

  1. [1]

    Achiam, D

    J. Achiam, D. Held, A. Tamar, and P. Abbeel. Constrained policy opti- mization. In International conference on machine learning, pages 22–

    work page

  2. [2]

    E. Altman. Constrained Markov decision processes. Routledge, 2021

    work page 2021

  3. [3]

    Arulkumaran, M

    K. Arulkumaran, M. P. Deisenroth, M. Brundage, and A. A. Bharath. Deep reinforcement learning: A brief survey. IEEE Signal Processing Magazine, 34(6):26–38, 2017

    work page 2017

  4. [4]

    Y . As, I. Usmanova, S. Curi, and A. Krause. Constrained policy opti- mization via bayesian world models. Preprint arXiv:2201.09802, 2022

    work page Pith review arXiv 2022

  5. [5]

    Bradbury, R

    J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclau- rin, G. Necula, A. Paszke, J. VanderPlas, S. Wanderman-Milne, and Q. Zhang. JAX: composable transformations of Python+NumPy pro- grams, 2018. URL http://github.com/jax-ml/jax

    work page 2018

  6. [6]

    Brunke, M

    L. Brunke, M. Greeff, A. W. Hall, Z. Yuan, S. Zhou, J. Panerati, and A. P. Schoellig. Safe learning in robotics: From learning-based control to safe reinforcement learning. Annual Review of Control, Robotics, and Autonomous Systems, 5(V olume 5, 2022):411–444", 2022

    work page 2022

  7. [7]

    X. Chen, C. Wang, Z. Zhou, and K. Ross. Randomized ensem- bled double q-learning: Learning fast without a model. Preprint arXiv:2101.05982, 2021

    arXiv 2021

  8. [8]

    Ganai, Z

    M. Ganai, Z. Gong, C. Yu, S. Herbert, and S. Gao. Iterative reacha- bility estimation for safe reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024

    work page 2024

Show all 31 references
  1. [9]

    S. Gu, L. Yang, Y . Du, G. Chen, F. Walter, J. Wang, and A. Knoll. A re- view of safe reinforcement learning: Methods, theory and applications. Preprint arXiv:2205.10330, 2024

    2024 arXiv

  2. [10]

    Haarnoja, A

    T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning , pages 1861–

  3. [11]

    Hafner, J

    D. Hafner, J. Pasukonis, J. Ba, and T. Lillicrap. Mastering diverse do- mains through world models. Preprint arXiv:2301.04104, 2023

    2023 arXiv

  4. [12]

    Hansen, H

    N. Hansen, H. Su, and X. Wang. Td-mpc2: Scalable, robust world mod- els for continuous control. In International Conference on Learning Representations (ICLR), 2024

    2024

  5. [13]

    Hewing, K

    L. Hewing, K. P. Wabersich, M. Menner, and M. N. Zeilinger. Learning- based model predictive control: Toward safe learning in control.Annual Review of Control, Robotics, and Autonomous Systems , 3(1):269–296, 2020

    2020

  6. [14]

    Huang, J

    W. Huang, J. Ji, B. Zhang, C. Xia, and Y . Yang. Safedreamer: Safe reinforcement learning with world models. In The Twelfth Interna- tional Conference on Learning Representations , 2024. URL https: //openreview.net/forum?id=tsE5HLYtYg

    2024

  7. [15]

    A. K. Jayant and S. Bhatnagar. Model-based safe deep reinforcement learning via a constrained proximal policy optimization algorithm. Ad- vances in Neural Information Processing Systems , 35:24432–24445, 2022

    2022

  8. [16]

    J. Ji, B. Zhang, J. Zhou, X. Pan, W. Huang, R. Sun, Y . Geng, Y . Zhong, J. Dai, and Y . Yang. Safety gymnasium: A unified safe reinforcement learning benchmark. In Thirty-seventh Conference on Neural Informa- tion Processing Systems Datasets and Benchmarks Track, 2023

    2023

  9. [18]

    J. Ji, J. Zhou, B. Zhang, J. Dai, X. Pan, R. Sun, W. Huang, Y . Geng, M. Liu, and Y . Yang. Omnisafe: An infrastructure for accelerating safe reinforcement learning research. Journal of Machine Learning Re- search, 25(285):1–6, 2024

    2024

  10. [19]

    T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y . Tassa, D. Sil- ver, and D. Wierstra. Continuous control with deep reinforcement learn- ing. Preprint arXiv:1509.02971, 2015

    2015 arXiv

  11. [21]

    Z. Liu, H. Zhou, B. Chen, S. Zhong, M. Hebert, and D. Zhao. Con- strained model-based reinforcement learning with robust cross-entropy method. arXiv preprint arXiv:2010.07968, 2020

    2010 arXiv

  12. [22]

    Z. Liu, Z. Cen, V . Isenbaev, W. Liu, S. Wu, B. Li, and D. Zhao. Con- strained variational policy optimization for safe reinforcement learning. In International Conference on Machine Learning, pages 13644–13668. PMLR, 2022

    2022

  13. [23]

    T. M. Moerland, J. Broekens, A. Plaat, C. M. Jonker, et al. Model- based reinforcement learning: A survey. Foundations and Trends® in Machine Learning, 16(1):1–118, 2023

    2023

  14. [24]

    R. R. Negenborn, B. De Schutter, M. A. Wiering, and H. Hellendoorn. Learning-based model predictive control for markov decision processes. IFAC Proceedings Volumes, 38(1):354–359, 2005

    2005

  15. [25]

    Nocedal and S

    J. Nocedal and S. J. Wright. Numerical Optimization. Springer, New York, NY , USA, second edition, 2006

    2006

  16. [26]

    M. L. Puterman. Markov decision processes: discrete stochastic dy- namic programming. John Wiley & Sons, 2014

    2014

  17. [27]

    A. Ray, J. Achiam, and D. Amodei. Benchmarking Safe Exploration in Deep Reinforcement Learning. 2019

    2019

  18. [28]

    Schulman, F

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Prox- imal policy optimization algorithms. Preprint arXiv:1707.06347, 2017

    2017 arXiv

  19. [29]

    Stooke, J

    A. Stooke, J. Achiam, and P. Abbeel. Responsive safety in reinforce- ment learning by pid lagrangian methods. In International Conference on Machine Learning, pages 9133–9143. PMLR, 2020

    2020

  20. [30]

    Usmanova, Y

    I. Usmanova, Y . As, M. Kamgarpour, and A. Krause. Log barriers for safe black-box optimization with application to safe reinforcement learning. Preprint arXiv:2207.10415, 2022

    2022 arXiv

  21. [31]

    R. Wei, N. Lambert, A. McDonald, A. Garcia, and R. Calandra. A uni- fied view on solving objective mismatch in model-based reinforcement learning. Preprint arXiv:2310.06253, 2023

    2023 arXiv

  22. [32]

    Wen and U

    M. Wen and U. Topcu. Constrained cross-entropy method for safe re- inforcement learning. Advances in Neural Information Processing Sys- tems, 31, 2018

    2018

  23. [33]

    Q. Yang, T. D. Simão, S. H. Tindemans, and M. T. Spaan. Wcsac: Worst-case soft actor critic for safety-constrained reinforcement learn- ing. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 10639–10646, 2021. A Algorithms Algorithm 1 SelectEl...

    2021

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.