Quantile fixed-point estimators for distributional policy evaluation achieve the parametric √n rate, attain the semiparametric efficiency bound for fixed m, remain efficient as m→∞, and admit Berry–Esseen inference.
Online statistical inference for nonlinear stochastic approximation with Markovian data.arXiv preprint arXiv:2302.07690, 2023a
3 Pith papers cite this work. Polarity classification is still indexing.
3
Pith papers citing it
abstract
We study the statistical inference of nonlinear stochastic approximation algorithms utilizing a single trajectory of Markovian data. Our methodology has practical applications in various scenarios, such as Stochastic Gradient Descent (SGD) on autoregressive data and asynchronous Q-Learning. By utilizing the standard stochastic approximation (SA) framework to estimate the target parameter, we establish a functional central limit theorem for its partial-sum process, $\boldsymbol{\phi}_T$. To further support this theory, we provide a matching semiparametric efficient lower bound and a non-asymptotic upper bound on its weak convergence, measured in the L\'evy-Prokhorov metric. This functional central limit theorem forms the basis for our inference method. By selecting any continuous scale-invariant functional $f$, the asymptotic pivotal statistic $f(\boldsymbol{\phi}_T)$ becomes accessible, allowing us to construct an asymptotically valid confidence interval. We analyze the rejection probability of a family of functionals $f_m$, indexed by $m \in \mathbb{N}$, through theoretical and numerical means. The simulation results demonstrate the validity and efficiency of our method.
representative citing papers
New Berry-Esseen bounds for multivariate martingale difference sequences achieve n^{-1/4} rate and polylog(d) dimension dependence in Kolmogorov distance.
Under nested local linearity, nonlinear two-time-scale SA achieves finite-time decoupled convergence; nonlinearity in the slow update alone can destroy it.
citing papers explorer
-
Statistical Efficiency and Inference of Quantile Distributional Reinforcement Learning
Quantile fixed-point estimators for distributional policy evaluation achieve the parametric √n rate, attain the semiparametric efficiency bound for fixed m, remain efficient as m→∞, and admit Berry–Esseen inference.
-
Berry-Esseen bounds for multivariate martingale difference sequences in the Kolmogorov distance
New Berry-Esseen bounds for multivariate martingale difference sequences achieve n^{-1/4} rate and polylog(d) dimension dependence in Kolmogorov distance.
-
Finite-Time Decoupled Convergence in Nonlinear Two-Time-Scale Stochastic Approximation
Under nested local linearity, nonlinear two-time-scale SA achieves finite-time decoupled convergence; nonlinearity in the slow update alone can destroy it.