Pith. sign in

REVIEW 4 major objections 4 minor 130 references

Adjustment Speed as a Safety Constraint for Nonstationary Reinforcement Learning

T0 review · 4 major / 4 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that safety under nonstationary reinforcement learning should be defined by whether the learning system can adapt fast enough, and proposes a feasibility gate that proactively tightens constraints when predicted change exc

desk verdict A novel adaptation-feasibility gate that is directionally supported by early-violation results but whose central capacity calibration is not yet valid. read the letter →

arxiv 2607.21646 v1 pith:TV4EPDYV submitted 2026-07-21 cs.LG

Timothy Tomashevskiy This is my paper
classification cs.LG
keywords nonstationaryreinforcementlearningadaptationfeasibilitysafetyconstraintcontextforecastingactionshieldingtransientrecoverycapacitysafeRL
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that when environments change, future states or regions become unsafe not because they are inherently dangerous, but because the adaptation needed to stay safe may lie outside the learning system's calibrated recovery envelope. It introduces a feasibility ratio that compares predicted context displacement (adaptation demand) with a measured recovery capacity, and triggers constraint tightening and action shielding when the ratio exceeds one. Experiments in a nonstationary driving domain show the approach primarily reduces safety violations in short-horizon windows just after context switches. This matters because it shifts safe RL from reacting to violations to anticipating whether adaptation itself is feasible.

What carries the argument

The central object is the adaptation-feasibility ratio ρ_t = ‖κ̂_{t+Δ} − κ̂_t‖₂/(C_adapt + ε), where κ̂_t is a learned latent-context embedding from a window of recent transitions, the numerator is the forecasted context displacement over a short horizon Δ, and C_adapt is a calibrated recovery-capacity threshold in the same latent units. This ratio converts a raw prediction of environmental change into a dimensionless trigger: values ≤1 indicate the shift is within the agent's safe-recovery envelope, while values >1 flag adaptation as infeasible and justify proactive safety intervention.

What would settle it

One concrete test would be to compute the realized forecast error e^pred_t = ‖κ̂_{t+Δ} − κ̂_obs_{t+Δ}‖₂ and show that the predicted context displacement A_t does not correlate with actual next-context displacement; if A_t carries no signal, the entire feasibility gate fires on noise. A second test would be to recalibrate C_adapt under the shielded, reward-adjusted policy and check whether the feasibility ratio ρ_t still separates safe from unsafe shifts; if it does not, the calibration is indeed stale.

Watch

Extended reading notes

Core claim

The central claim is that the rate at which the environment changes, relative to the agent's empirically calibrated ability to recover safe behavior, is itself a safety constraint. Concretely, the paper defines adaptation demand A_t as the Euclidean distance between the predicted future latent context and the current context, and adaptation capacity C_adapt as a quantile of past context displacements after which the agent historically recovered below a violation threshold. When the ratio ρ_t = A_t/(C_adapt+ε) exceeds 1, the framework tightens the admissible action threshold and activates an action-level shield, replacing unsafe actions with lower-risk alternatives. The empirical result is a

Load-bearing premise

The calibrated recovery threshold C_adapt is measured from training rollouts of the unmodified policy, but it is then used to trigger a shield and reward penalties that change the deployed policy, so the measured 'achievable adaptation capacity' may no longer reflect the system that is actually running.

Editorial extensions

If this is right

  • If the paper is right, safe RL under nonstationarity should be augmented with a monitoring layer that predicts adaptation demand and gates safety interventions, rather than assuming the policy can always recover.
  • The feasibility ratio provides an actionable, calibration-based criterion for when to tighten constraints and activate shielding, which can be added on top of existing safe-RL methods.
  • The ablation shows that optimization-level adjustment (a reward penalty) and action-level shielding are complementary: shielding alone suppresses peak and tail risk, while the combination reduces early switch-conditioned violations.
  • The approach implies a shift from static safety specifications to feasibility-driven, time-varying safety constraints that depend on the learning system's current adaptation capability.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If adaptation capacity is policy-dependent, the calibrated C_adapt may become stale once the shield and reward adjustment change the deployed policy, so a practical extension is to recalibrate the threshold online or maintain a distribution over recovery capacities.
  • The same feasibility gate could be used in continual learning to decide when to pause learning, roll back parameters, or invoke a more conservative controller—not merely to filter actions.
  • A testable extension would be to evaluate whether the method generalizes to continuous action spaces and richer safety costs, where the shield's fallback behavior may need to be a learned safe controller rather than a discrete lowest-risk action.
  • The empirical comparison suggests a research direction: comparing feasibility-gated proactive shielding against reactive safety filters that act only after an unsafe action is proposed, to quantify the value of anticipation itself.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes Adjustment Speed as a Safety Constraint in Nonstationary Reinforcement Learning (ASASC-NS). The method learns a latent context representation from transition windows, forecasts the future context over a short horizon, and measures adaptation demand as the Euclidean displacement between current and predicted context embeddings. A recovery-capacity threshold C_adapt is calibrated from training rollouts as a quantile of context displacements after which the agent recovered below a tolerated violation rate. When the feasibility ratio ρ_t = A_t/(C_adapt+ε) exceeds 1, the method tightens the admissible action set and activates an action-level shield; it also optionally applies a reward penalty. Experiments on a nonstationary highway-env merge-v0 driving task report that the Full method reduces EARLYVIOL relative to baseline, while Shield-only reduces PEAKRISK and TAILVIOL. The paper frames the contribution as a proactive, feasibility-based safety mechanism for nonstationary RL.

Significance. If the central mechanism is validated, the paper introduces a new and useful principle—defining safety through adaptation feasibility—and provides a concrete, implementable trigger for proactive intervention. Strengths include a transparent algorithmic description, clearly separated ablations, confidence intervals on the headline metrics, and an explicit statement of scope and limitations. The conditional theoretical characterization is honest in its modesty. However, the empirical support is currently limited to one environment and one RL backbone, and the calibration of C_adapt—the load-bearing component of the method—is not validated against the behavior it is supposed to predict. The significance of the contribution depends on closing that gap.

major comments (4)
  1. [§4.3, Eq. (7), App. C.1] C_adapt is calibrated as a quantile of A_t only among shifts whose post-shift average violation was below η. Conditioning on the outcome means that the threshold is not an estimate of the feasible/infeasible boundary: large-A_t episodes that recovered by chance are included, and small-A_t episodes that failed are excluded. Consequently, ρ_t > 1 need not indicate true infeasibility, and ρ_t ≤ 1 can include infeasible shifts. Lemma 2's proof sketch ('By construction, C_adapt...') makes this circularity explicit. No recovery-rate-vs-A_t curve or equivalent calibration diagnostic is reported to show that the threshold separates recoverable from non-recoverable displacements. This is load-bearing for the central claim that the trigger identifies adaptation infeasibility. Please report such a curve, or at minimum show that trigger times align with periods of elevated post-switch violation risk
  2. [§4.4–4.5, Algorithm 1] C_adapt is calibrated on unmodified training rollouts, but it is then used to gate reward shaping and shielding, which change the deployed policy and therefore the distribution of A_t and the actual recovery envelope. The same threshold is used for the shielded policy without recalibration or sensitivity analysis. Since the empirical comparison is between methods that differ precisely in these interventions, a fixed C_adapt may be miscalibrated for the shielded policy, confounding the attribution of the safety improvements to the adaptation-feasibility signal. Please report the sensitivity of the main results to the calibration hyperparameters (q, η, λ, β_AS) or provide an adaptive/re-calibrated version of C_adapt.
  3. [Table 1 vs §5.4] The ablation summary claims that 'optimization-level adjustment contributes most directly to improved early adaptation safety.' The reported CIs do not support this attribution: Adj-only EARLYVIOL is 0.0560±0.0218, which overlaps with baseline 0.0690±0.0069. The Full method achieves 0.0440±0.0103, but it also includes shielding, so the specific contribution of the optimization-level adjustment alone is not established. Please either soften the claim or provide a paired significance test or effect-size analysis separating the two interventions.
  4. [App. C.3] The appendix states that the context-prediction diagnostic e_pred_t is logged to check that the adaptation-demand signal reflects actual context evolution, but no numerical values or plots are reported. Since the validity of A_t (Eq. 6) as a measure of adaptation demand is load-bearing for the entire framework, the absence of this diagnostic leaves the central premise unverified. Please report e_pred_t (mean, distribution, or time series) and show that forecast errors are small enough to support the interpretation of A_t.
minor comments (4)
  1. [Table 2] Several key hyperparameters (q, η, β_AS, λ, m, L) are listed as 'validation-selected' without giving the selected values. For reproducibility, please provide concrete values or a link to a configuration file.
  2. [Fig. 2] The text claims confidence intervals are available, but Figure 2 shows no error bars or shaded regions. Please add them or state explicitly that they are omitted.
  3. [General] There are several minor spacing/formatting issues, e.g., 'ADJUSTMENTSPEED' in the title and inconsistent use of 'C adapt' vs 'C_adapt'. Also, the phrase in App. A about avoiding 'a second, inconsistent proof' is unusual; consider rewording.
  4. [Lemma 2 / App. A.3] Lemma 2 is essentially a restatement of the definition of C_adapt. If kept, it should be presented as an operational definition rather than a lemma, to avoid the appearance of a substantive guarantee.

Circularity Check

1 steps flagged · score 6.0 of 10

The infeasibility 'prediction' is a threshold test against a fitted quantile; Lemma 2 restates the definition of C_adapt, while the Table 1 safety evaluation remains an independent empirical check.

  1. self definitional [§4.3 Eq. (7)–(8); Appendix A.3 Lemma 2 proof]
    "Cadapt = Quantileq({At : ¯vt:t+Hrec ≤η}) ... By construction, C_adapt is calibrated from rollouts in which the agent recovered below a target violation level after context shifts of comparable magnitude."

    The feasibility ratio ρ_t = A_t/(C_adapt+ε) is presented as predicting 'adaptation-infeasible' when A_t > C_adapt. But Eq. 7 defines C_adapt as a quantile of A_t values from rollouts that already recovered below the tolerated violation level. Lemma 2 then 'proves' that A_t ≤ C_adapt means within the safe-recovery envelope and A_t > C_adapt means outside it. This is true by construction: it restates the calibration rule instead of deriving capacity from independent evidence. The threshold is a fitted quantile of the very quantity it classifies. The Table 1 safety metrics are not forced by this definition, so the circularity is partial and confined to the theoretical characterization.

full rationale

The paper contains no self-citations and the empirical protocol (Table 1, ablations, nonstationarity sweep) is a genuine out-of-sample comparison: the safety metrics are not the calibration objective. However, the paper's conditional theory collapses to a definition: the 'calibrated recovery capacity' C_adapt is exactly a quantile of recoverable adaptation-demand values, and Lemma 2's feasibility claim restates this definition ('By construction'). Thus the 'prediction' that a shift is adaptation-infeasible is a threshold comparison against a fitted value, not an independent estimate of the agent's true recovery envelope. A further concern—calibration under the unmodified policy while the gate is used under a shielded policy—affects validity/staleness rather than circularity, and the missing quantitative forecast-error diagnostic (Appendix C.3) is a reporting gap, not a circular step. Overall, the central empirical contribution survives, but the theoretical framing includes a definitional step presented as a proof, warranting a partial-circularity score of 6.

Assumptions & free parameters 8 free parameters · 5 assumptions · 0 invented entities

No new physical entities are postulated. The latent context κ is a learned representation of a standard construct in context-RL, not an invented entity with a falsifiable handle. The paper's 8 free parameters (all fitted or hand-chosen) and 5 domain assumptions carry the entire load; the central safety decision ρ_t > 1 is a comparison against the fitted threshold C_adapt in a learned space whose semantics are unverified.

free parameters (8)
  • C_adapt (recovery capacity threshold) = not reported (quantile of recoverable context-displacements)
    Eq. 7: calibrated from training rollouts as Quantile_q of A_t over shifts with post-shift violation rate ≤ η. The entire trigger ρ_t = A_t/(C_adapt+ε) is a comparison against this fitted value, so the method's behavior depends directly on it.
  • Quantile q = not reported
    Table 2: 'high quantile of recoverable shifts'; controls trigger conservatism and is selected on calibration rollouts; sensitivity to q is not reported.
  • Tolerance η = not reported
    Table 2: 'validation-selected target violation level'; defines what counts as a recoverable shift during calibration.
  • Adjustment penalty β_AS = not reported
    Eq. 9: validation-selected strength of the optimization-level safety penalty in the TD target.
  • Tightening strength λ = not reported
    Eq. 11: validation-selected slope mapping feasibility ratio into threshold tightening.
  • Context window length m and context history length L = not reported
    Table 2: both 'validation-selected'; define the transition window h_t and the embedding history fed to the forecaster.
  • Forecast horizon Δ = 10 context-update steps
    §4.2: fixed short horizon chosen by hand ('we use Δ = 10'); A_t and ρ_t depend on it, and no sensitivity analysis is given.
  • Lipschitz constant L (Lemma 1) = unquantified
    Appendix A.2: the bound P(v_t=1) ≤ α + Lδ_t depends on 'L captures the sensitivity of the safety event to context change'; never measured, so the lemma is untestable as stated.
assumptions (5)
  • domain assumption Environment is governed by a latent context variable z_t such that transitions depend on (s_t, a_t, z_t); in experiments, nonstationarity is switch-like Markov switching.
    §3 and §5.1: M_t = (S,A,P_t,R_t) with latent context and p_stay switching. If real nonstationarity is gradual, continuous, or adversarial, the calibrated capacity and window metrics may not transfer.
  • domain assumption Transition kernel and safety cost are Lipschitz continuous with respect to context.
    Appendix A.2 Lemma 1 assumes this; no verification or estimation of the Lipschitz constant is provided.
  • domain assumption Safety-cost estimator ĉ ranks candidate actions consistently with realized violation risk up to bounded estimation error.
    Appendix A.4 Proposition 1 is conditional on this; in the driving experiments ĉ is built from collision/overlap, gap, and time-to-collision diagnostics, and the shield's improvements inherit this assumption.
  • ad hoc to paper Euclidean displacement in the learned context embedding measures actual adaptation demand, and C_adapt fitted in the same space from baseline rollouts remains valid for the shielded policy.
    §4.3 Eqs. 6–8: the whole gate compares two quantities in a learned space whose distance-to-recovery-difficulty calibration is asserted; the stated validation diagnostic (Appendix C.3, e^pred_t) is never reported numerically.
  • domain assumption Recovery behavior observed on training rollouts is representative of recovery behavior during deployment.
    Calibration of C_adapt (Eq. 7) assumes historical recoverability transfers forward; the policy modification introduced by shield/adjustment (§4.4–4.5) means the calibrated capacity is for a different policy than the one being gated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Adjustment Speed as a Safety Constraint for Nonstationary Reinforcement Learning." pith.science (2026). https://pith.science/paper/TV4EPDYV

@misc{pith2026260721646,
  author       = {Pith},
  title        = {Pith review of: Adjustment Speed as a Safety Constraint for Nonstationary Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TV4EPDYV}},
  note         = {Machine review of arXiv:2607.21646}
}
read the original abstract

Ensuring safety in reinforcement learning under nonstationarity requires determining whether a learning system can safely adapt to forecasted environmental change within the required recovery horizon. Existing safe reinforcement learning methods typically assume stationary environments and do not explicitly consider adaptation speed as a safety concern. However, when environments evolve over time, delayed adaptation may result in transient unsafe behavior. This paper proposes adjustment speed as a safety constraint for nonstationary reinforcement learning. The central idea is to define safety in terms of adaptation feasibility: future states or regions may become unsafe when the adaptation required to remain safe exceeds the learning system's calibrated recovery capacity. The proposed framework uses learned context representations and short-horizon context forecasts to estimate adaptation demand and compare it with the agent's achievable adaptation capacity. When predicted adaptation demand exceeds the calibrated recovery capacity, the framework proactively tightens the admissible action set and activates an action-level shield to reduce unsafe behavior before violations occur. Experiments in a nonstationary driving environment show that the proposed approach primarily reduces safety violations in short-horizon windows aligned with context changes. Ablation studies further show that shielding is more conservative for peak- and tail-risk suppression, while optimization-level adjustment provides additional reductions in short-horizon switch-conditioned violations. These results support adaptation feasibility as a practical safety principle for reinforcement learning under nonstationarity and demonstrate that proactive intervention can improve safety during periods of environmental change.

Figures

Figures reproduced from arXiv: 2607.21646 by the authors.

Figure 1
Figure 1. Conceptual illustration of adaptation feasibility. The red curve denotes the required adaptation rate in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Effect of nonstationarity on transient safety. Early-phase violation rate (EARLYVIOL) as a function of the context-persistence probability pstay. Lower values of pstay correspond to more frequent context switches and stronger nonstationarity. The proposed method main￾tains lower violation rates across the displayed switching regimes, although the improvement is not monotonic and the gap varies across settings [PITH… view at source ↗
Figure 5
Figure 5. Cumulative mean violation rate during train [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

130 extracted references · 13 linked inside Pith

  1. [1]

    doi:10.1109/ms.2018.4321239 , number =

    David Lorge Parnas , title =. doi:10.1109/ms.2018.4321239 , number =

    arXiv 2018

  2. [2]

    1999 , publisher =

    Constrained Markov decision processes , author =. 1999 , publisher =

    1999

  3. [3]

    2018 , publisher =

    Reinforcement learning: An introduction , author =. 2018 , publisher =

    2018

  4. [4]

    arXiv preprint arXiv:2004.07584 , year =

    Reinforcement learning for safety-critical control under model uncertainty, using control lyapunov functions and control barrier functions , author =. arXiv preprint arXiv:2004.07584 , year =

    arXiv 2004

  5. [5]

    2016 IEEE 55th Conference on Decision and Control (CDC) , pages =

    Safe learning of regions of attraction for uncertain, nonlinear systems with Gaussian processes , author =. 2016 IEEE 55th Conference on Decision and Control (CDC) , pages =. 2016 , organization =

    2016

  6. [6]

    arXiv preprint arXiv:2205.10330 , year =

    A review of safe reinforcement learning: Methods, theory and applications , author =. arXiv preprint arXiv:2205.10330 , year =

    arXiv

  7. [7]

    arXiv preprint arXiv:2006.10701 , year =

    Deep reinforcement learning amidst lifelong non-stationarity , author =. arXiv preprint arXiv:2006.10701 , year =

    arXiv 2006

  8. [8]

    International Conference on Machine Learning (ICML) , year =

    Constrained Policy Optimization , author =. International Conference on Machine Learning (ICML) , year =

Show all 130 references
  1. [9]

    Risk-Sensitive and Robust Decision-Making: a

    Chow, Yinlam and Ghavamzadeh, Mohammad and Janson, Lucas and Pavone, Marco , booktitle =. Risk-Sensitive and Robust Decision-Making: a

  2. [10]

    arXiv preprint arXiv:1502.02267 , year =

    Policy Gradient for Coherent Risk Measures , author =. arXiv preprint arXiv:1502.02267 , year =

    arXiv

  3. [11]

    Journal of Machine Learning Research , volume =

    A Comprehensive Survey on Safe Reinforcement Learning , author =. Journal of Machine Learning Research , volume =

  4. [12]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    Safe Model-based Reinforcement Learning with Stability Guarantees , author =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  5. [13]

    1998 , publisher =

    Reinforcement learning: An introduction , author =. 1998 , publisher =

    1998

  6. [14]

    2015 European Control Conference (ECC) , pages =

    Safe and robust learning control with Gaussian processes , author =. 2015 European Control Conference (ECC) , pages =. 2015 , organization =

    2015

  7. [15]

    International Conference on Machine Learning , pages =

    Robust multi-objective bayesian optimization under input noise , author =. International Conference on Machine Learning , pages =. 2022 , organization =

    2022

  8. [16]

    Neural networks , volume =

    Continual lifelong learning with neural networks: A review , author =. Neural networks , volume =. 2019 , publisher =

    2019

  9. [17]

    Automated machine learning: methods, systems, challenges , pages =

    Meta-learning , author =. Automated machine learning: methods, systems, challenges , pages =. 2019 , publisher =

    2019

  10. [18]

    International conference on machine learning , pages =

    Pac-inspired option discovery in lifelong reinforcement learning , author =. International conference on machine learning , pages =. 2014 , organization =

    2014

  11. [19]

    International conference on machine learning , pages =

    Policy and value transfer in lifelong reinforcement learning , author =. International conference on machine learning , pages =. 2018 , organization =

    2018

  12. [20]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume =

    Lifelong learning with a changing action set , author =. Proceedings of the AAAI Conference on Artificial Intelligence , volume =

  13. [21]

    arXiv preprint arXiv:1710.03641 , year =

    Continuous adaptation via meta-learning in nonstationary and competitive environments , author =. arXiv preprint arXiv:1710.03641 , year =

    arXiv

  14. [22]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume =

    Provably efficient primal-dual reinforcement learning for cmdps with non-stationary objectives and constraints , author =. Proceedings of the AAAI Conference on Artificial Intelligence , volume =

  15. [23]

    Advances in Neural Information Processing Systems , volume =

    Towards safe policy improvement for non-stationary MDPs , author =. Advances in Neural Information Processing Systems , volume =

  16. [24]

    arXiv preprint arXiv:2003.00660 , year =

    Upper confidence primal-dual optimization: Stochastically constrained markov decision processes with adversarial losses and unknown transitions , author =. arXiv preprint arXiv:2003.00660 , year =

    2003 arXiv

  17. [25]

    International Conference on Machine Learning , pages =

    Optimizing for the future in non-stationary mdps , author =. International Conference on Machine Learning , pages =. 2020 , organization =

    2020

  18. [26]

    International Conference on Machine Learning , pages =

    Safe policy search for lifelong reinforcement learning with sublinear regret , author =. International Conference on Machine Learning , pages =. 2015 , organization =

    2015

  19. [27]

    International Conference on Artificial Intelligence and Statistics , pages =

    Provably efficient model-free algorithms for non-stationary cmdps , author =. International Conference on Artificial Intelligence and Statistics , pages =. 2023 , organization =

    2023

  20. [28]

    2021 IEEE International Conference on Robotics and Automation (ICRA) , pages =

    Context-aware safe reinforcement learning for non-stationary environments , author =. 2021 IEEE International Conference on Robotics and Automation (ICRA) , pages =. 2021 , organization =

    2021

  21. [29]

    2022 IEEE 61st Conference on Decision and Control (CDC) , pages =

    Finite-time complexity of online primal-dual natural actor-critic algorithm for constrained Markov decision processes , author =. 2022 IEEE 61st Conference on Decision and Control (CDC) , pages =. 2022 , organization =

    2022

  22. [30]

    arXiv preprint arXiv:2405.16601 , year =

    A CMDP-within-online framework for meta-safe reinforcement learning , author =. arXiv preprint arXiv:2405.16601 , year =

    arXiv

  23. [31]

    arXiv preprint arXiv:2111.00552 , year =

    Policy optimization for constrained mdps with provable fast global convergence , author =. arXiv preprint arXiv:2111.00552 , year =

    arXiv

  24. [32]

    Machine Learning , volume =

    All-time safety and sample-efficient meta update for online safe meta reinforcement learning under Markov task transition , author =. Machine Learning , volume =. 2025 , publisher =

    2025

  25. [33]

    IEEE Journal of Selected Topics in Signal Processing , volume =

    Online convex optimization in dynamic environments , author =. IEEE Journal of Selected Topics in Signal Processing , volume =. 2015 , publisher =

    2015

  26. [34]

    Proceedings of the AAAI conference on artificial intelligence , volume =

    Safe online convex optimization with unknown linear safety constraints , author =. Proceedings of the AAAI conference on artificial intelligence , volume =

  27. [35]

    IEEE Transactions on Cybernetics , volume =

    Adaptive safe reinforcement learning with full-state constraints and constrained adaptation for autonomous vehicles , author =. IEEE Transactions on Cybernetics , volume =. 2023 , publisher =

    2023

  28. [36]

    International Conference on Machine Learning , pages =

    Enforcing hard constraints with soft barriers: Safe reinforcement learning in unknown stochastic environments , author =. International Conference on Machine Learning , pages =. 2023 , organization =

    2023

  29. [37]

    IEEE Transactions on Pattern Analysis and Machine Intelligence , year =

    A review of safe reinforcement learning: Methods, theories and applications , author =. IEEE Transactions on Pattern Analysis and Machine Intelligence , year =

  30. [38]

    The 30th international joint conference on artificial intelligence (ijcai) , year =

    Policy learning with constraints in model-free reinforcement learning: A survey , author =. The 30th international joint conference on artificial intelligence (ijcai) , year =

  31. [39]

    arXiv preprint arXiv:2402.02025 , year =

    A survey of constraint formulations in safe reinforcement learning , author =. arXiv preprint arXiv:2402.02025 , year =

    arXiv

  32. [40]

    ACM Computing Surveys (CSUR) , volume =

    A survey of reinforcement learning algorithms for dynamically varying environments , author =. ACM Computing Surveys (CSUR) , volume =. 2021 , publisher =

    2021

  33. [41]

    Journal of Artificial Intelligence Research , volume =

    Towards continual reinforcement learning: A review and perspectives , author =. Journal of Artificial Intelligence Research , volume =

  34. [42]

    Machine Learning , volume =

    A taxonomy for similarity metrics between markov decision processes , author =. Machine Learning , volume =. 2022 , publisher =

    2022

  35. [43]

    Proceedings of the AAAI conference on artificial intelligence , volume =

    Safe reinforcement learning via shielding under partial observability , author =. Proceedings of the AAAI conference on artificial intelligence , volume =

  36. [44]

    Machine learning , volume =

    Bayesian optimization with safety constraints: safe and automatic parameter tuning in robotics , author =. Machine learning , volume =. 2023 , publisher =

    2023

  37. [45]

    2022 , school =

    Reinforcement Learning for Non-stationary problems , author =. 2022 , school =

    2022

  38. [46]

    arXiv preprint arXiv:1611.05763 , year =

    Learning to reinforcement learn , author =. arXiv preprint arXiv:1611.05763 , year =

    arXiv

  39. [47]

    System Modeling and Optimization: Proceedings of the 10th IFIP Conference New York City, USA, August 31--September 4, 1981 , pages =

    The Bayesian approach to global optimization , author =. System Modeling and Optimization: Proceedings of the 10th IFIP Conference New York City, USA, August 31--September 4, 1981 , pages =. 2005 , organization =

    1981

  40. [48]

    Foundations and Trends

    Bayesian reinforcement learning: A survey , author =. Foundations and Trends. 2015 , publisher =

    2015

  41. [49]

    Journal of Global Optimization , volume =

    Bayesian heuristic approach to global optimization and examples , author =. Journal of Global Optimization , volume =. 2002 , publisher =

    2002

  42. [50]

    2021 , publisher =

    Constrained Markov decision processes , author =. 2021 , publisher =

    2021

  43. [51]

    IEEE Transactions on Automatic Control , volume =

    Risk-constrained Markov decision processes , author =. IEEE Transactions on Automatic Control , volume =. 2014 , publisher =

    2014

  44. [52]

    International Conference on Machine Learning , pages =

    Safe reinforcement learning in constrained markov decision processes , author =. International Conference on Machine Learning , pages =. 2020 , organization =

    2020

  45. [53]

    International Workshop on the Foundations of Trustworthy AI Integrating Learning, Optimization and Reasoning , pages =

    Safe learning and optimization techniques: Towards a survey of the state of the art , author =. International Workshop on the Foundations of Trustworthy AI Integrating Learning, Optimization and Reasoning , pages =. 2020 , organization =

    2020

  46. [54]

    IEEE transactions on pattern analysis and machine intelligence , volume =

    Meta-learning in neural networks: A survey , author =. IEEE transactions on pattern analysis and machine intelligence , volume =. 2021 , publisher =

    2021

  47. [55]

    Annual Review of Control, Robotics, and Autonomous Systems , volume =

    Safe learning in robotics: From learning-based control to safe reinforcement learning , author =. Annual Review of Control, Robotics, and Autonomous Systems , volume =. 2022 , publisher =

    2022

  48. [56]

    Journal of mathematics and mechanics , pages =

    A Markovian decision process , author =. Journal of mathematics and mechanics , pages =. 1957 , publisher =

    1957

  49. [57]

    Journal of mathematical analysis and applications , volume =

    Optimal control of Markov processes with incomplete state information I , author =. Journal of mathematical analysis and applications , volume =. 1965 , publisher =

    1965

  50. [58]

    IJCAI: proceedings of the conference , volume =

    Hidden parameter markov decision processes: A semiparametric regression approach for discovering latent task parametrizations , author =. IJCAI: proceedings of the conference , volume =

  51. [59]

    Sequence learning: paradigms, algorithms, and applications , pages =

    Hidden-mode markov decision processes for nonstationary sequential decision making , author =. Sequence learning: paradigms, algorithms, and applications , pages =. 2001 , publisher =

    2001

  52. [60]

    Artificial intelligence , volume =

    Planning and acting in partially observable stochastic domains , author =. Artificial intelligence , volume =. 1998 , publisher =

    1998

  53. [61]

    Proceedings of the fifth international joint conference on Autonomous agents and multiagent systems , pages =

    Improving reinforcement learning with context detection , author =. Proceedings of the fifth international joint conference on Autonomous agents and multiagent systems , pages =

  54. [62]

    Handbooks in operations research and management science , volume =

    Markov decision processes , author =. Handbooks in operations research and management science , volume =. 1990 , publisher =

    1990

  55. [63]

    Advances in neural information processing systems , volume =

    Bayes-adaptive pomdps , author =. Advances in neural information processing systems , volume =

  56. [64]

    2002 , publisher =

    Optimal Learning: Computational procedures for Bayes-adaptive Markov decision processes , author =. 2002 , publisher =

    2002

  57. [65]

    2019 , school =

    Safe exploration in reinforcement learning: Theory and applications in robotics , author =. 2019 , school =

    2019

  58. [66]

    International workshop on hybrid systems: Computation and control , pages =

    Safety verification of hybrid systems using barrier certificates , author =. International workshop on hybrid systems: Computation and control , pages =. 2004 , organization =

    2004

  59. [67]

    Mitigating Distribution Shifts: Uncertainty-Aware Offline-to-Online Reinforcement Learning , author =

  60. [68]

    2019 American Control Conference (ACC) , pages =

    Safety-aware reinforcement learning framework with an actor-critic-barrier structure , author =. 2019 American Control Conference (ACC) , pages =. 2019 , organization =

    2019

  61. [69]

    IEEE Transactions on robotics , volume =

    Barrier-certified adaptive reinforcement learning with applications to brushbot navigation , author =. IEEE Transactions on robotics , volume =. 2019 , publisher =

    2019

  62. [70]

    International conference on machine learning , pages =

    Model-agnostic meta-learning for fast adaptation of deep networks , author =. International conference on machine learning , pages =. 2017 , organization =

    2017

  63. [71]

    International conference on machine learning , pages =

    Online meta-learning , author =. International conference on machine learning , pages =. 2019 , organization =

    2019

  64. [72]

    Foundations and Trends

    Introduction to online convex optimization , author =. Foundations and Trends. 2016 , publisher =

    2016

  65. [73]

    Advances in Neural Information Processing Systems , volume =

    Meta-reinforcement learning with universal policy adaptation: Provable near-optimality under all-task optimum comparator , author =. Advances in Neural Information Processing Systems , volume =

  66. [74]

    International Conference on Machine Learning , pages =

    Memory efficient online meta learning , author =. International Conference on Machine Learning , pages =. 2021 , organization =

    2021

  67. [75]

    Advances in neural information processing systems , volume =

    Adaptive gradient-based meta-learning methods , author =. Advances in neural information processing systems , volume =

  68. [76]

    International Conference on Machine Learning , pages =

    Crpo: A new approach for safe reinforcement learning with convergence guarantee , author =. International Conference on Machine Learning , pages =. 2021 , organization =

    2021

  69. [77]

    International conference on machine learning , pages =

    Efficient off-policy meta-reinforcement learning via probabilistic context variables , author =. International conference on machine learning , pages =. 2019 , organization =

    2019

  70. [78]

    2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages =

    Improved Robustness and Safety for Pre-Adaptation of Meta Reinforcement Learning with Prior Regularization , author =. 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages =. 2022 , organization =

    2022

  71. [79]

    2.3 softmax units for multinoulli output distributions , author =

    6.2. 2.3 softmax units for multinoulli output distributions , author =. Deep learning , volume =. 2016 , publisher =

    2016

  72. [80]

    arXiv preprint arXiv:2206.04436 , year =

    Towards safe reinforcement learning via constraining conditional value-at-risk , author =. arXiv preprint arXiv:2206.04436 , year =

    arXiv

  73. [81]

    Journal of Advances in Information Fusion , volume =

    A constrained POMDP formulation and algorithmic solution for radar resource management in multi-target tracking , author =. Journal of Advances in Information Fusion , volume =. 2021 , publisher =

    2021

  74. [82]

    Advances in neural information processing systems , volume =

    Non-stationary Markov decision processes, a worst-case approach using model-based reinforcement learning , author =. Advances in neural information processing systems , volume =

  75. [83]

    Proceedings of the 2016 international conference on autonomous agents & multiagent systems , pages =

    Measuring the distance between finite markov decision processes , author =. Proceedings of the 2016 international conference on autonomous agents & multiagent systems , pages =

    2016

  76. [84]

    arXiv preprint arXiv:2103.04706 , year =

    A taxonomy of similarity metrics for markov decision processes , author =. arXiv preprint arXiv:2103.04706 , year =

    arXiv

  77. [85]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    Safe learning of regions of attraction for uncertain, nonlinear systems with gaussian processes , author =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  78. [86]

    Mathematics of Operations Research , volume =

    Robust dynamic programming , author =. Mathematics of Operations Research , volume =. 2005 , publisher =. doi:10.1287/moor.1040.0135 , url =

    2005

  79. [87]

    Operations Research , volume =

    Robust control of Markov decision processes with uncertain transition matrices , author =. Operations Research , volume =. 2005 , publisher =. doi:10.1287/opre.1050.0243 , url =

    2005

  80. [88]

    Advances in Neural Information Processing Systems , year =

    Safe exploration in finite Markov decision processes with Gaussian processes , author =. Advances in Neural Information Processing Systems , year =

  81. [89]

    Robotics: Science and Systems (RSS) , year =

    Safety and Liveness Guarantees through Reach-Avoid Reinforcement Learning , author =. Robotics: Science and Systems (RSS) , year =. doi:10.15607/RSS.2021.XVII.077 , note =

    2021 doi

  82. [90]

    Proceedings of the 39th International Conference on Machine Learning , series =

    Reachability Constrained Reinforcement Learning , author =. Proceedings of the 39th International Conference on Machine Learning , series =. 2022 , publisher =

    2022

  83. [91]

    Proceedings of the AAAI Conference on Artificial Intelligence , year =

    Safe Reinforcement Learning via Shielding under Partial Observability , author =. Proceedings of the AAAI Conference on Artificial Intelligence , year =

  84. [92]

    Proceedings of the 7th Annual Learning for Dynamics & Control Conference , series =

    Back to Base: Towards Hands-Off Learning via Safe Resets with Reach-Avoid Safety Filters , author =. Proceedings of the 7th Annual Learning for Dynamics & Control Conference , series =. 2025 , publisher =

    2025

  85. [93]

    Proceedings of the 7th Annual Learning for Dynamics & Control Conference , series =

    Safe Learning in the Real World via Adaptive Shielding with Hamilton-Jacobi Reachability , author =. Proceedings of the 7th Annual Learning for Dynamics & Control Conference , series =. 2025 , publisher =

    2025

  86. [94]

    Proceedings of the 31st AAAI Conference on Artificial Intelligence (AAAI) , pages =

    Lyapunov-based Safe Policy Optimization for Continuous Control , author =. Proceedings of the 31st AAAI Conference on Artificial Intelligence (AAAI) , pages =. 2017 , url =

    2017

  87. [95]

    Proceedings of the 31st Conference on Neural Information Processing Systems (NeurIPS) , year =

    Inverse Reward Design , author =. Proceedings of the 31st Conference on Neural Information Processing Systems (NeurIPS) , year =

  88. [96]

    ACM Computing Surveys , volume =

    A Review on Safe Reinforcement Learning: Methods, Theory and Applications , author =. ACM Computing Surveys , volume =. 2022 , publisher =. doi:10.1145/3459990 , url =

    2022 doi

  89. [97]

    International Conference on Learning Representations (ICLR) , year =

    VariBAD: A Very Good Method for Bayes-Adaptive Deep Reinforcement Learning , author =. International Conference on Learning Representations (ICLR) , year =

  90. [98]

    International Conference on Learning Representations (ICLR) , year =

    Learning to Adapt in Dynamic, Real-World Environments through Meta-Reinforcement Learning , author =. International Conference on Learning Representations (ICLR) , year =

  91. [99]

    IEEE Transactions on Neural Networks and Learning Systems , volume =

    Risk-Sensitive Reinforcement Learning Algorithms , author =. IEEE Transactions on Neural Networks and Learning Systems , volume =. 2014 , publisher =. doi:10.1109/TNNLS.2013.2295538 , url =

    2014

  92. [100]

    International Conference on Learning Representations (ICLR) , year =

    Reward Constrained Policy Optimization , author =. International Conference on Learning Representations (ICLR) , year =

  93. [101]

    Advances in Neural Information Processing Systems , year =

    A Lyapunov-based Approach to Safe Reinforcement Learning , author =. Advances in Neural Information Processing Systems , year =

  94. [102]

    European Journal of Control , year =

    Control Barrier Functions: Theory and Applications , author =. European Journal of Control , year =

  95. [103]

    International Conference on Algorithmic Learning Theory , year =

    On Upper-Confidence Bound Policies for Non-Stationary Bandit Problems , author =. International Conference on Algorithmic Learning Theory , year =

  96. [104]

    arXiv preprint arXiv:1707.06347 , year =

    Proximal Policy Optimization Algorithms , author =. arXiv preprint arXiv:1707.06347 , year =

    arXiv

  97. [105]

    2013 , publisher =

    Adaptive Control , author =. 2013 , publisher =

    2013

  98. [106]

    1998 , publisher =

    Essentials of Robust Control , author =. 1998 , publisher =

    1998

  99. [107]

    Advances in Neural Information Processing Systems , year =

    Robust Reinforcement Learning , author =. Advances in Neural Information Processing Systems , year =

  100. [108]

    International Conference on Machine Learning , year =

    Robust Adversarial Reinforcement Learning , author =. International Conference on Machine Learning , year =

  101. [109]

    International Conference on Machine Learning , year =

    Scaling Up Robust MDPs Using Function Approximation , author =. International Conference on Machine Learning , year =

  102. [110]

    AAAI Conference on Artificial Intelligence , year =

    Distributionally Robust Reinforcement Learning , author =. AAAI Conference on Artificial Intelligence , year =

  103. [111]

    Advances in Neural Information Processing Systems , year =

    Deep Reinforcement Learning in a Handful of Trials Using Probabilistic Dynamics Models , author =. Advances in Neural Information Processing Systems , year =

  104. [112]

    International Conference on Machine Learning , year =

    Online Convex Programming and Generalized Infinitesimal Gradient Ascent , author =. International Conference on Machine Learning , year =

  105. [113]

    Operations Research , year =

    Non-Stationary Stochastic Optimization , author =. Operations Research , year =

  106. [114]

    International Conference on Machine Learning , year =

    On Calibration of Modern Neural Networks , author =. International Conference on Machine Learning , year =

  107. [115]

    Advances in Neural Information Processing Systems , year =

    Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles , author =. Advances in Neural Information Processing Systems , year =

  108. [116]

    Nature , year =

    Human-level control through deep reinforcement learning , author =. Nature , year =

  109. [117]

    ICLR , year =

    Continuous control with deep reinforcement learning , author =. ICLR , year =

  110. [118]

    Proceedings of the National Academy of Sciences , year =

    Overcoming Catastrophic Forgetting in Neural Networks , author =. Proceedings of the National Academy of Sciences , year =

  111. [119]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    Risk-Constrained Reinforcement Learning with Percentile Risk Criteria , author =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  112. [120]

    International Conference on Machine Learning (ICML) , year =

    Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor , author =. International Conference on Machine Learning (ICML) , year =

  113. [121]

    International Conference on Machine Learning , year=

    Contextual Markov Decision Processes , author=. International Conference on Machine Learning , year=

  114. [122]

    Proceedings of the AAAI Conference on Artificial Intelligence , year =

    Safe Reinforcement Learning via Shielding , author =. Proceedings of the AAAI Conference on Artificial Intelligence , year =

  115. [123]

    Proceedings of the 35th International Conference on Machine Learning , series =

    Gal Dalal and Roi Gilboa and Shie Mannor and Ran El-Yaniv , title =. Proceedings of the 35th International Conference on Machine Learning , series =. 2018 , editor =

    2018

  116. [124]

    IEEE Transactions on Automatic Control , year =

    A Predictive Safety Filter for Learning-Based Control of Constrained Nonlinear Dynamical Systems , author =. IEEE Transactions on Automatic Control , year =

  117. [125]

    International Conference on Machine Learning , year =

    Meta Reinforcement Learning as Task Inference , author =. International Conference on Machine Learning , year =

  118. [126]

    Advances in Neural Information Processing Systems , year =

    Lifelong Reinforcement Learning with Context-Dependent Exploration , author =. Advances in Neural Information Processing Systems , year =

  119. [127]

    International Conference on Learning Representations , year =

    Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learning , author =. International Conference on Learning Representations , year =

  120. [128]

    Nature , year=

    Loss of Plasticity in Deep Continual Learning , author=. Nature , year=

  121. [129]

    Conference on Lifelong Learning Agents , year=

    Loss of Plasticity in Continual Deep Reinforcement Learning , author=. Conference on Lifelong Learning Agents , year=

  122. [130]

    Advances in Neural Information Processing Systems , year=

    A Study of Plasticity Loss in On-Policy Deep Reinforcement Learning , author=. Advances in Neural Information Processing Systems , year=

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.