REVIEW 3 cited by
An Analysis of Quantile Temporal-Difference Learning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
An Analysis of Quantile Temporal-Difference Learning
read the original abstract
We analyse quantile temporal-difference learning (QTD), a distributional reinforcement learning algorithm that has proven to be a key component in several successful large-scale applications of reinforcement learning. Despite these empirical successes, a theoretical understanding of QTD has proven elusive until now. Unlike classical TD learning, which can be analysed with standard stochastic approximation tools, QTD updates do not approximate contraction mappings, are highly non-linear, and may have multiple fixed points. The core result of this paper is a proof of convergence to the fixed points of a related family of dynamic programming procedures with probability 1, putting QTD on firm theoretical footing. The proof establishes connections between QTD and non-linear differential inclusions through stochastic approximation theory and non-smooth analysis.
Forward citations
Cited by 3 Pith papers
-
Auditing the Risk Claims of Distributional Reinforcement Learning
40-95% of the strongest risk trade-off claims of QR-DQN, C51 and IQN are refuted; the learned risk is a training artifact, not real environment stochasticity.
-
Statistical Efficiency and Inference of Quantile Distributional Reinforcement Learning
Quantile fixed-point estimators of return distributions attain the parametric √n rate and the semiparametric efficiency bound for fixed and diverging numbers of quantiles, with a Berry–Esseen guarantee for smooth functionals.
-
Statistical Efficiency and Inference of Quantile Distributional Reinforcement Learning
Quantile fixed-point estimators for distributional policy evaluation achieve the parametric √n rate, attain the semiparametric efficiency bound for fixed m, remain efficient as m→∞, and admit Berry–Esseen inference.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.