Continuous-time reinforcement learning: ellipticity enables model-free value function approximation

Summary. The paper studies off-policy reinforcement learning for controlling continuous-time Markov diffusion processes observed and acted upon at discrete times. It introduces a model-free approach with function approximation that learns value and advantage functions directly from data. Leveraging the uniform ellipticity of the diffusion matrix, the authors establish Hilbert-space positive-definiteness and boundedness properties for the associated Bellman operators. These properties are used to analyze the Sobolev-prox fitted q-learning algorithm, which iteratively solves least-squares regression problems. Oracle inequalities are derived for the estimation error, with leading terms given by the best approximation error of the function classes, their localized complexity, an exponentially decaying optimization error, and a numerical discretization error. The central conclusion is that ellipticity renders reinforcement learning with function approximation for these processes statistically no harder than supervised learning.

Significance. If the oracle inequalities and the underlying operator coercivity hold under the stated ellipticity assumption, the work supplies a clean theoretical reduction of a continuous-time RL problem to standard nonparametric regression rates. This is significant because it isolates ellipticity as a structural condition that removes the usual horizon- or discount-dependent blow-up in Bellman-operator norms, thereby placing continuous-time diffusion control on the same statistical footing as supervised learning. The explicit accounting for discretization and optimization errors is also practically useful for algorithm design.