Introduces a learned arrow of time in MDPs that aligns with the Jordan-Kinderlehrer-Otto notion for stochastic processes and enables practical RL utilities like reachability and side-effect detection.
Safe exploration in markov decision processes
4 Pith papers cite this work. Polarity classification is still indexing.
4
Pith papers citing it
abstract
In environments with uncertain dynamics exploration is necessary to learn how to perform well. Existing reinforcement learning algorithms provide strong exploration guarantees, but they tend to rely on an ergodicity assumption. The essence of ergodicity is that any state is eventually reachable from any other state by following a suitable policy. This assumption allows for exploration algorithms that operate by simply favoring states that have rarely been visited before. For most physical systems this assumption is impractical as the systems would break before any reasonable exploration has taken place, i.e., most physical systems don't satisfy the ergodicity assumption. In this paper we address the need for safe exploration methods in Markov decision processes. We first propose a general formulation of safety through ergodicity. We show that imposing safety by restricting attention to the resulting set of guaranteed safe policies is NP-hard. We then present an efficient algorithm for guaranteed safe, but potentially suboptimal, exploration. At the core is an optimization formulation in which the constraints restrict attention to a subset of the guaranteed safe policies and the objective favors exploration policies. Our framework is compatible with the majority of previously proposed exploration methods, which rely on an exploration bonus. Our experiments, which include a Martian terrain exploration problem, show that our method is able to explore better than classical exploration methods.
citation-role summary
background 2
citation-polarity summary
roles
background 2polarities
background 2representative citing papers
The paper categorizes five concrete AI safety problems arising from flawed objectives, costly evaluation, and learning dynamics.
SBSRL approximates worst-case safety optimization over uncertain dynamics via finite sampling, adds epistemic-uncertainty-constrained exploration, and supplies high-probability safety guarantees plus finite-time sample-complexity bounds for near-optimal policies.
PF-CD3Q uses online particle filtering to estimate fatigue parameters and constrains a deep Q-learning agent to solve fatigue-aware human-robot task planning as a CMDP.
citing papers explorer
-
Learning the Arrow of Time
Introduces a learned arrow of time in MDPs that aligns with the Jordan-Kinderlehrer-Otto notion for stochastic processes and enables practical RL utilities like reachability and side-effect detection.
-
Concrete Problems in AI Safety
The paper categorizes five concrete AI safety problems arising from flawed objectives, costly evaluation, and learning dynamics.
-
Sampling-Based Safe Reinforcement Learning
SBSRL approximates worst-case safety optimization over uncertain dynamics via finite sampling, adds epistemic-uncertainty-constrained exploration, and supplies high-probability safety guarantees plus finite-time sample-complexity bounds for near-optimal policies.
-
Safe reinforcement learning with online filtering for fatigue-predictive human-robot task planning and allocation in production
PF-CD3Q uses online particle filtering to estimate fatigue parameters and constrains a deep Q-learning agent to solve fatigue-aware human-robot task planning as a CMDP.