Policy gradient formulas are derived for random, policy-dependent time horizons, and the corrected state-space factor (expected runtime) materially improves convergence in experiments.
Fr\'{e}chet derivatives of expected functionals of solutions to stochastic differential equations
1 Pith paper cite this work. Polarity classification is still indexing.
abstract
In the analysis of stochastic dynamical systems described by stochastic differential equations (SDEs), it is often of interest to analyse the sensitivity of the expected value of a functional of the solution of the SDE with respect to perturbations in the SDE parameters. In this paper, we consider path functionals that depend on the solution of the SDE up to a stopping time. We derive formulas for Fr\'{e}chet derivatives of the expected values of these functionals with respect to bounded perturbations of the drift, using the Cameron-Martin-Girsanov theorem for the change of measure. Using these derivatives, we construct an example to show that the map that sends the change of drift to the corresponding relative entropy is not in general convex. We then analyse the existence and uniqueness of solutions to stochastic optimal control problems defined on possibly random time intervals, as well as gradient-based numerical methods for solving such problems.
fields
cs.LG 1years
2025 1verdicts
ACCEPT 1representative citing papers
citing papers explorer
-
Reinforcement Learning with Random Time Horizons
Policy gradient formulas are derived for random, policy-dependent time horizons, and the corrected state-space factor (expected runtime) materially improves convergence in experiments.