REVIEW 3 major objections 5 minor 1 cited by
Universal Approximation Theorem of Deep Q-Networks
T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read A residual-network DQN can match any continuous optimal Q-function on compact sets.
desk verdict The main approximation theorem is invalid as stated because the DQN never sees time, and the convergence theorem is a conditional repackaging of known results. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The argument runs on three objects. The DQN itself is a residual network whose hidden layers update $x^{(l+1)} = x^{(l)} + h_{\theta_l}(x^{(l)}, a)\Delta t$, the Euler discretization of an ODE, which lets the proof invoke residual-network universal approximation on compact sets. A large-deviation bound on the martingale-driven state process supplies the compact set $K_R$ carrying probability at least $1-\delta$. For convergence, the Bellman operator $T(Q)(t,s,a) = r(t,s,a) + \gamma\mathbb{E}[\max_{a'} Q(t+\Delta t, s', a') \mid s,a]$ is shown to be a $\gamma$-contraction in the sup norm, so its fixed point is $Q^*$, and the averaged stochastic-approximation ODE $\dot{\theta} = \bar{H}(\theta)$ is steered to that fixed point by Lyapunov-style gradient conditions. Viscosity solutions of the HJB equation for $V^*$ play a supporting role in justifying the regularity of the value function without requiring classical differentiability of $Q^*$.
What would settle it
Let $Q^*(t,s,a) = e^{t}g(s,a)$ for a nonzero Lipschitz $g$ on $[0,T] \times S \times A$. Any DQN whose input is $(s,a)$ only is a function independent of $t$, so its uniform distance from $Q^*$ is at least $(e^T - 1)/2$ times the maximum of $|g|$ over the compact region; a concrete check is whether any trained network achieves $\sup |Q_\theta - Q^*| < \epsilon$ for $\epsilon$ smaller than that bound, which would refute the approximation theorem as stated.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is Theorem 3.1: under Lipschitz regularity on the MDP coefficients and continuity of $Q^*$, for every $\epsilon > 0$ and every $\delta \in (0,1)$ there exists a residual-block DQN $Q_\theta$ such that $\sup_{K_R}|Q_\theta(t,s,a) - Q^*(t,s,a)| < \epsilon$, where $K_R = [0,T] \times \{s : \|s\| \le R_1\} \times A$ is a compact set that the trajectory occupies with probability at least $1-\delta$. Theorem 3.8 adds that the stochastic-approximation Q-learning iterates satisfy $\|Q_{\theta_k} - Q^*\|_\infty \to 0$ almost surely under ergodic sampling, Robbins-Monro learning rates, representability, identifiability, and a gradient/Bellman-error correlation assumption. The two theorems together say that DQNs inherit the universal approximation capacity of ResNets for continuous $Q^*$, and that a natural training loop can in principle find the approximating parameters.
Load-bearing premise
The central load-bearing premise is that a residual network whose inputs are only state and action can uniformly approximate $Q^*(t,s,a)$ on $[0,T] \times S \times A$ even though $Q^*$ depends on the remaining time horizon $t$; the paper's approximation proof applies a network approximation lemma stated for compact subsets of $\mathbb{R}^{n+m}$ to the $(1+n+m)$-dimensional set $[0,T] \times S \times A$ without supplying $t$ as an input.
Editorial extensions
If this is right
- A DQN with enough residual layers and parameters can match any continuous optimal Q-function uniformly on the high-probability region of state-action space, so expressiveness is not the bottleneck in continuous-time DQN learning.
- The depth choice $L=N$ ties network capacity to time discretization: refining the time grid naturally suggests deeper networks, and error $\epsilon$ would require depth scaling roughly like $\epsilon^{-\kappa}$ under stronger smoothness assumptions.
- Under representability, identifiable parametrization, and the negative-correlation gradient condition, Q-learning with target values converges to the true optimal Q-function almost surely, not merely to a stationary point.
- The Bellman operator remains a contraction in the continuous-time discretized setting, so the classical Q-learning fixed-point logic survives the passage to martingale-driven SDEs.
- Approximation error is controlled only on compact sets with high probability; the large-deviation radius $R_1$ absorbs the tail risk, so unbounded state spaces are handled probabilistically rather than uniformly.
Reading between the lines
- The paper leaves implicit that its network receives only $(s,a)$, not the time coordinate $t$; a fully honest statement of the approximation claim would require either feeding $t$ as an input or restricting to $Q^*$ with negligible time variation, and checking that gap is a natural next test.
- One testable extension is to compare, on finite-horizon benchmark control tasks, the empirical approximation error of state-action-only DQNs versus networks with $t$ as an input; the paper's theorem predicts both can work in principle, while the gap would reveal how much time-dependence matters.
- If combined with quantitative approximation rates for residual networks, the $L=N$ coupling suggests measurable scaling laws—depth as a function of $1/\epsilon$ and of the modulus of continuity of $Q^*$—that could be validated on the 1D control experiment included in the paper.
- The convergence proof's reliance on representability and the negative-correlation condition suggests that in practice, target networks and experience replay may be doing the work of enforcing these assumptions, and failures of convergence could be diagnosed as violations of Assumption 3.6 rather than of the Bellman contraction.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper develops a continuous-time framework for Deep Q-Networks, modeling the environment as an SDE driven by a continuous square-integrable martingale and connecting DQNs to residual networks and FBSDEs. The main theoretical claims are: (i) Theorem 3.1, that a residual-block DQN of the form in Definition 2.4 can uniformly approximate the optimal Q-function Q*(t,s,a) on a compact set K_R with high probability; and (ii) Theorem 3.8, that a Q-learning parameter update (22) converges almost surely to Q* in the supremum norm under assumptions 3.3-3.6. The paper includes an appendix with proofs and a short numerical study of a 1D control task.
Significance. If it were correct, the paper would provide a clean universal approximation statement for DQNs in a continuous-time stochastic control setting and a convergence guarantee for a Q-learning variant, thereby bridging deep RL and stochastic control. The framing around compact high-probability sets, large deviation bounds, and viscosity solutions is reasonable in spirit, and the residual-network approximation route is a recognized method. However, the central approximation claim is invalid for the architecture as defined, because the network has no time input while Q* is time-dependent, and the convergence theorem is largely a restatement of assumptions whose main dissipativity condition is postulated rather than proved. The paper does not provide machine-checked proofs or a reproducible code repository; the numerical appendix describes experiments but no code or pretrained artifacts are included.
major comments (3)
- [Definition 2.4, Remark 2.5, and Appendix A.6] The network defined in Definition 2.4 does not receive time as an input. Equation (2) sets x^(0)_k = (s_{t_k}, a_k) and updates x^(l+1)_k = x^(l)_k + h_{θ_l}(x^(l)_k, a_k)Δt, with no t_k appearing in the recursion. For a fixed parameter vector θ, Q_θ(t,s,a) is therefore constant in t. Yet the target Q*(t,s,a) in Definition 2.13 depends on t through the remaining horizon; even in a degenerate MDP with h=σ=0, r=1, g=0 and a singleton action set, Q*(t,s,a) = (1-e^{-γ(T-t)})/γ is t-dependent. Theorem 3.1 asserts a uniform approximation over K_R = [0,T] x S_R x A in Eq. (21). This cannot hold for ε smaller than half the range of Q* in t. Appendix A.6 applies Lemma 2.8 to K_R, a compact subset of R^{1+n+m}, but Lemma 2.8 is stated for compact subsets of R^{n+m}; enlarging the ambient dimension does not repair the argument because the parametrized class of functions is a proper subclass of continuous functions, namely those independent of t. Moreover, Definition 2.4 only defines Q_θ at the grid times t_k, not for arbitrary (t,s,a), so the supremum in (21) is not even well-defined unless an interpolation is specified. Theorem 3.1 is therefore invalid as stated.
- [Appendix A.7, Eq. (41)] The proof of Theorem 3.8 conflates the discrete-time Bellman fixed point with the continuous-time Q* of Definition 2.13. Equation (41) defines TQ = r + γ E[max_{a'} Q(t+Δt, s', a') | s,a], which is a discrete-time Bellman operator with discount γ per step. The continuous-time optimal Q-function, however, involves the discounted integral e^{-γ(u-t)} r du and the terminal term e^{-γ(T-t)} g(s_T); Eq. (13) itself only states that the discrete Bellman relation holds approximately as Δt -> 0. The claim 'the Bellman operator has a unique fixed point Q*' is not justified for this continuous-time Q*, and hence the convergence of θ_k to the parameter representing Q* does not follow from a contraction argument on the discrete operator. This is a load-bearing gap in the proof of the central convergence result.
- [Assumptions 3.5-3.6 and Appendix A.7] Theorem 3.8 relies on assumptions that essentially postulate the conclusion. Assumption 3.5(i) assumes Q* is exactly representable in the network class, and Assumption 3.6(ii) assumes a negative-correlation inequality between the function error and the Bellman error. In the Lyapunov analysis of Appendix A.7, the proof computes the derivative of V(θ)=1/2||Q_θ-Q*||^2_{L^2(μ)}, writes 'Relating this directly to Assumption 3.6 (ii) requires further steps...', and then simply asserts that Assumption 3.6(ii) implies the dynamics drive θ to θ*. No derivation from the network structure, the Bellman operator, and the sampling measure is given. The theorem also claims convergence in the supremum norm over [0,T]×S×A, while the assumptions and Lyapunov function are formulated in a weighted L^2(μ) norm; the passage from θ_k -> θ* to ||Q_{θ_k}-Q*||_∞ -> 0 is not established, since continuity on a compact set plus large deviation bounds do not give uniform convergence on the whole, unbounded state space. As a result, Theorem 3.8 is not proven beyond restating the dissipativity condition.
minor comments (5)
- [Lemma 2.8 proof, Appendix A.3] The proof treats vector-valued approximation by approximating each component with a separate residual block and then stacking outputs. This is acceptable for a concatenated-output network, but it does not show that a single residual-block family of the form A_l η(B_l x + b_l) with a single activation layer can approximate an arbitrary vector-valued continuous function; that is a standard vector-valued UAT statement and should be stated and cited as such.
- [Lemma 2.10 proof, Appendix A.4] The proof asserts a specific exponential large-deviation bound with explicit constants C1, C2, C3, but it does not prove that bound or give a reference that applies to SDEs driven by general continuous square-integrable martingales. Since this lemma is used only to select R1, a weaker qualitative statement with a proper reference would be cleaner and avoids the current hand-constructed constants.
- [Remark 3.2] The remark suggests a rate L ∝ (1/ε)^κ (e.g., κ=2 for an Euler-Maruyama interpretation), but no rate theorem is proved or cited for the specific residual-block architecture of Definition 2.4. This is presented as speculation and should be explicitly marked as such.
- [Theorem 3.8 statement] The update rule in Eq. (22)-(23) omits the Δt scaling of the reward term that appears in the Section 2.4 target y_k = r Δt + e^{-γΔt} max Q; the convergence theorem uses the unscaled version, which is inconsistent with the continuous-time approximation described earlier.
- [General presentation] There are numerous typos and spacing artifacts from the source text, such as 'timet', 'It ˆo', and broken reference formatting. These are cosmetic but should be cleaned before any resubmission.
Circularity Check
Theorem 3.8's convergence is largely assumed rather than derived: the negative-correlation Lyapunov condition is postulated on exactly the error being driven to zero, and the sup-norm proof explicitly offers 'assume uniform convergence holds over the entire space' as an escape hatch.
-
self definitional
[Section 3.2, Assumption 3.6(ii), and Appendix A.7 (Proof of Theorem 3.8)]
"Assumption 3.6 (ii) (Negative Correlation Condition) There exists a constant c>0 such that for all θ∈Θ: ∫∫_{S×A} (Qθ(t,s,a)−Q∗(t,s,a))δθ(t,s,a)||∇θQθ(t,s,a)||^2 μ(ds,da) ≤ −c||Qθ−Q∗||^2_{μ,G}. ... 'if we accept that Assumption 3.6 captures the necessary conditions for stability, it implies that the dynamics ˙θ=¯H(θ) drive θ towards θ∗ ... Thus, under Assumptions 3.5 and 3.6, θ∗ is the unique and globally asymptotically stable equilibrium point of the ODE.'"
Assumption 3.6(ii) is a Lyapunov descent inequality stated directly on the squared error ||Qθ−Q∗||^2_{μ,G}: it postulates that the expected TD-update direction, weighted by ∇θQθ, is negatively correlated with the error Qθ−Q∗. That is precisely the mechanism the convergence proof must establish. The proof does not derive stability from the condition; it says that if one 'accepts' the condition, it 'implies' the desired drift toward θ∗. Combined with Assumption 3.5(i), which simply postulates that Q∗ is exactly representable as Qθ∗ in the network class, the theorem's convergence conclusion is effectively a restatement of the assumptions rather than a derived result.
-
self definitional
[Appendix A.7, final paragraph (Proof of Theorem 3.8)]
"To strengthen this to uniform convergence (L∞), we can argue that the convergence is uniform on the compact set [0,T]×KS×A ... because continuous functions on compact sets are uniformly continuous. We combine this with the large deviation bounds (Lemma 2.10) or assume uniform convergence holds over the entire space under the given assumptions. Thus, lim k→∞ ||Qθk−Q∗||∞ = 0, almost surely."
The theorem's conclusion is exactly the statement lim_k ||Qθk−Q∗||∞ = 0 over [0,T]×S×A. The proof offers two routes: a compact-set argument plus large deviations, or 'assume uniform convergence holds over the entire space.' The second route simply imports the theorem's conclusion as an unstated hypothesis; none of Assumptions 3.3–3.6 or 2.1–2.16 asserts uniform convergence over the whole unbounded state space. The claimed sup-norm convergence is therefore not derived from the stated assumptions but is assumed directly.
full rationale
The approximation theorem (Theorem 3.1) is not circular: it imports an external residual-network universal approximation theorem (Lemma 2.8, attributed to Li et al. 2022 and standard UAT) and an external large-deviation bound, with no load-bearing self-citation. The time-input mismatch between Definition 2.4 and the time-dependent Q∗ of Definition 2.13 is a serious correctness defect in the proof as written, but it is not a circularity, because the paper does not redefine the target in terms of the network. The circularity is concentrated in Theorem 3.8. Assumption 3.6(ii) is a negative-correlation inequality posed directly on (Qθ−Q∗) and δθ, i.e., a Lyapunov descent condition on exactly the error the theorem promises to drive to zero; Appendix A.7 then says convergence follows if one 'accepts' that condition rather than deriving it. More explicitly, the sup-norm proof offers as one route 'assume uniform convergence holds over the entire space,' which is the theorem's conclusion imported as an unstated assumption. With Assumption 3.5(i) also postulating exact representability of Q∗, the convergence theorem's central claim is close to a restatement of its assumptions. Self-citations to Qi (2024) appear only as contextual remarks and do not affect either theorem. Overall circularity score: 7.
Assumptions & free parameters
assumptions (8)
- ad hoc to paper Q* is continuous on [0,T] x S x A (Assumption 2.3).
- standard math Residual networks can uniformly approximate any continuous function on compact sets (Lemma 2.8, citing Li et al. 2022).
- standard math Large deviation bound for SDEs driven by continuous square-integrable martingales (Lemma 2.10, citing Dembo 2009 and Mao 2007).
- domain assumption The state-action sampling process is ergodic with a unique invariant distribution and continuous transition probabilities (Assumption 3.3(i)).
- ad hoc to paper Q* is exactly representable by the network and identifiable near the optimum (Assumption 3.5).
- ad hoc to paper Negative correlation condition between Bellman error and function error (Assumption 3.6(ii)).
- domain assumption Activation function is non-linear, non-constant, and Lipschitz; parameter space is compact (Assumption 2.16).
- standard math Learning rates satisfy the Robbins-Monro conditions (Assumption 3.4).
Cite this review
Pith. "Pith review of Universal Approximation Theorem of Deep Q-Networks." pith.science (2026). https://pith.science/paper/64UVOOEL
@misc{pith2026250502288,
author = {Pith},
title = {Pith review of: Universal Approximation Theorem of Deep Q-Networks},
year = {2026},
howpublished = {\url{https://pith.science/paper/64UVOOEL}},
note = {Machine review of arXiv:2505.02288}
}
abstract
We establish a continuous-time framework for analyzing Deep Q-Networks (DQNs) via stochastic control and Forward-Backward Stochastic Differential Equations (FBSDEs). Considering a continuous-time Markov Decision Process (MDP) driven by a square-integrable martingale, we analyze DQN approximation properties. We show that DQNs can approximate the optimal Q-function on compact sets with arbitrary accuracy and high probability, leveraging residual network approximation theorems and large deviation bounds for the state-action process. We then analyze the convergence of a general Q-learning algorithm for training DQNs in this setting, adapting stochastic approximation theorems. Our analysis emphasizes the interplay between DQN layer count, time discretization, and the role of viscosity solutions (primarily for the value function $V^*$) in addressing potential non-smoothness of the optimal Q-function. This work bridges deep reinforcement learning and stochastic control, offering insights into DQNs in continuous-time settings, relevant for applications with physical systems or high-frequency data.
Figures
Forward citations
Cited by 1 Pith paper
-
Universal Approximation Theorem for Deep Q-Learning via FBSDE System
A conditional universal approximation theorem shows that residual operator-block DQNs with depth aligned to Bellman iterations can approximate the optimal Q-function, assuming a neural operator class with controlled L...
Reference graph
Works this paper leans on
-
[1]
Continuity of the Target Function: Assumption 2.3 posits that the optimal Q-functionQ∗(t,s,a ) is continuous on its domain [0,T ]×S× A. This continuity is essential because standard universal approximation theorems apply to continuous functions
-
[2]
Relevance of Compact Sets: Real-world or simulated processes often evolve within bounded regions, or their analysis can be restricted to such regions with high probability. Lemma 2.10, under the linear growth conditions of Assumption 2.2, guarantees that for any desired probability 1−δ, we can find boundsR1,R 2 such that the state-action trajectory (st,at...
-
[3]
Universal Approximation on Compact Sets: Lemma 2.8 states the universal approximation property for the chosen class of networks (specifically, networks built from the residual blocks hθl). It asserts that such networks can approximate any continuous function uniformly on a compact set likeKR to any desired precisionϵ
-
[4]
Synthesis: SinceQ∗ is assumed continuous (Point 1) and the relevant behavior of the system occurs within the compact setKR with high probability (Point 2), the universal approximation capability of the residual network architecture on KR (Point 3) ensures that there exists a set of parameters θ for the DQNQθ such thatQθ is arbitrarily close toQ∗ uniformly...
work page 2022
-
[5]
Dynamic Programming Principle: Under suitable assumptions, the value functionV∗ satisfies the following dynamic programming principle: V∗(t,s ) = sup π∈Π E "Z t+∆t t e−γ(u−t)r(u,su,π (u,su))du +e−γ∆tV∗(t + ∆t,st+∆t)|st =s # , for any small ∆t >0. This principle states that the optimal value starting at time t and states can be obtained by optimizing over ...
-
[6]
Without loss of generality, we can assume that V∗(t0,s 0) = ϕ(t0,s 0)
Viscosity Subsolution: Letϕ∈C1,2([0,T ]×S ) be a smooth test function such thatV∗−ϕ has a local maximum at (t0,s 0)∈ [0,T )×S . Without loss of generality, we can assume that V∗(t0,s 0) = ϕ(t0,s 0). By the dynamic programming principle, for any fixed constant controlau =a∈A over [t0,t 0 + ∆t], we have: V∗(t0,s 0)≥ E "Z t0+∆t t0 e−γ(u−t0)r(u,su,a )du +e−γ∆...
-
[7]
Viscosity Supersolution: The proof for the supersolution is analogous. Let ϕ∈C1,2([0,T ]×S ) be a smooth test function such thatV∗−ϕ has a local minimum at (t0,s 0)∈ [0,T )×S andV∗(t0,s 0) =ϕ(t0,s 0). By the dynamic programming principle, there exists anϵ-optimal control sequence. Taking limits appropriately and using Itˆo’s formula leads to the supersolu...
work page 2006
-
[8]
The sequence{αk} satisfies the Robbins-Monro conditions (Assumption 3.4). (Condition (C1))
Show all 17 references
-
[9]
This is ensured by Assumption 3.3 (i))
The sequence{(sk,ak)} is an ergodic Markov process with a unique invariant distributionµ(ds,da ), and the empirical measure of the samples converges weakly toµ almost surely. This is ensured by Assumption 3.3 (i)). (Condition (C2))
-
[10]
This will be shown below
The functionH is continuous inθ and Lipschitz continuous in (t,s,a,s ′), uniformly inθ. This will be shown below. (Condition (C3))
-
[11]
(Condition (C5))
The target Q-values yk are uniformly bounded (Assumption 3.3 (ii)). (Condition (C5))
-
[12]
This is guaranteed by the compactness of Θ (Assumption 2.16 (i))
The sequenceθk is bounded with probability one. This is guaranteed by the compactness of Θ (Assumption 2.16 (i)). (Condition (C6)) We also need to verify condition (C4), which requires us to show the existence and uniqueness of a globally asymptotically stable equilibrium poin...
1995
-
[13]
Baseline: Default parameters, 2 residual blocks (RES BLOCKS=2)
-
[14]
High LR: Learning rate increased to 1× 10−3
-
[15]
Fewer ResBlocks: No residual blocks used (RES BLOCKS=0), reducing the network to a standard MLP
-
[16]
High Noise: Environment noise increased toσ = 0.3
-
[17]
For reproducibility and fair comparison, a fixed random seed (SEED=42) is used across all runs
Slow Target Update: Target network updated every 500 steps. For reproducibility and fair comparison, a fixed random seed (SEED=42) is used across all runs. B.2. Results The training performance comparing the different configurations is presented in Figure 1 (assuming figure ge...
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.