{"id":"5a345f35-6237-414b-b610-df9df64e4c04","arxiv_id":"2505.02288","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"A DQN built from residual blocks inherits universal approximation for continuous Q-functions, but the claimed convergence proof assumes the Lyapunov condition and exact representability that essentially assert the result.","lead":"This paper claims that Deep Q-Networks can approximate the optimal value function in continuous-time control to any desired accuracy, and that a Q-learning training procedure converges to it. A generalist might read it to see whether deep reinforcement learning theory can borrow tools from stochastic analysis, but the proofs lean on assumptions that carry most of the load.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The DQN defined in Definition 2.4 has no time input, so it cannot uniformly approximate the time-dependent Q* on [0,T] x S x A; Theorem 3.1 fails as stated.","rationale":"The reader's weakest-assumption diagnosis is correct and is the most load-bearing issue in the paper. The architecture in Definition 2.4 is time-autonomous, while the target Q* in Definition 2.13 depends on t through the remaining horizon; no amount of residual depth or width can remove that mismatch. The proof of Theorem 3.1 in Appendix A.6 simply invokes Lemma 2.8 on a domain of the wrong dimension and, more importantly, on a function class that does not contain all continuous functions on [0,T] x S x A. The explicit counterexample with h=sigma=0, r=1, g=0 shows the theorem is false as stated. Theorem 3.8 has independent, serious gaps: Assumptions 3.5 and 3.6 essentially assert the desired conclusion through exact representability and a negative-correlation Lyapunov condition, which the paper itself in Remark 3.7 calls strong and open to verify, and the proof's Lyapunov step is conceded to need further steps. Additionally, the discrete Bellman operator in Eq. (41) is not shown to have Q* from Eq. (8) as its fixed point. Appendix B is not independent experimental support: it describes figures 'assuming figure generated by the code' and outcomes based on 'typical results,' so no actual runs are reported. These considerations reinforce the rejection without changing the verdict.","tokens_in":23529,"tokens_out":5867,"duration_ms":77599,"concrete_test":"Formalize the counterexample: take S=R (or any open set), A={a0}, h=0, sigma=0, r(t,s,a)=1, g=0, gamma=1, T=1. Then Q*(t,s,a0)=1-e^{-(1-t)}, and Assumptions 2.1-2.3 and 2.16 hold. Write any network from Definition 2.4 symbolically; since the recursion contains no t_k term, its output is identical for t=0 and t=1 for every fixed (s,a0). Compute sup_{t in [0,1]} |Q_theta(t,s,a0) - Q*(t,s,a0)| for any such constant-in-t output; show it is at least (1-e^{-1})/2 approx 0.316, contradicting the epsilon < 0.316 case of Theorem 3.1. A second check: append t as an input feature to the residual blocks and re-run the same construction; the universal approximation argument then goes through, confirming that the missing time coordinate, not the UAT, is the failure point.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Definition 2.4 constructs Q_theta from residual blocks with x^(0)_k = (s_{t_k}, a_k) and x^(l+1)_k = x^(l)_k + h_{theta_l}(x^(l)_k, a_k) Delta t. No coordinate t_k appears anywhere in the recursion, and Remark 2.5 confirms the network takes only the current state and action as input. Hence, for each fixed theta, the function (t,s,a) -> Q_theta(t,s,a) is constant in t. But Q* in Definition 2.13 is genuinely time-dependent: the remaining horizon T-t enters the discounted integral and the terminal term. For a degenerate MDP with h=sigma=0, r=1, g=0, and a singleton action set, Q*(t,s,a) = (1 - e^{-gamma(T-t)})/gamma, which varies with t. Appendix A.6 applies Lemma 2.8 to K_R, a subset of R^{1+n+m}, even though the lemma is stated for a compact subset of R^{n+m}; extending the universal approximation theorem to R^{1+n+m} would not repair the argument because the parametrized class is a proper subclass of continuous functions, namely those independent of t. Consequently the sup over t in [0,T] in Eq. (21) cannot be made small for epsilon below half the range of Q* in t, so Theorem 3.1 is invalid for the architecture as defined.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper develops a continuous-time framework for Deep Q-Networks, modeling the environment as an SDE driven by a continuous square-integrable martingale and connecting DQNs to residual networks and FBSDEs. The main theoretical claims are: (i) Theorem 3.1, that a residual-block DQN of the form in Definition 2.4 can uniformly approximate the optimal Q-function Q*(t,s,a) on a compact set K_R with high probability; and (ii) Theorem 3.8, that a Q-learning parameter update (22) converges almost surely to Q* in the supremum norm under assumptions 3.3-3.6. The paper includes an appendix with proofs and a short numerical study of a 1D control task.","tokens_in":23815,"tokens_out":4893,"duration_ms":67103,"significance":"If it were correct, the paper would provide a clean universal approximation statement for DQNs in a continuous-time stochastic control setting and a convergence guarantee for a Q-learning variant, thereby bridging deep RL and stochastic control. The framing around compact high-probability sets, large deviation bounds, and viscosity solutions is reasonable in spirit, and the residual-network approximation route is a recognized method. However, the central approximation claim is invalid for the architecture as defined, because the network has no time input while Q* is time-dependent, and the convergence theorem is largely a restatement of assumptions whose main dissipativity condition is postulated rather than proved. The paper does not provide machine-checked proofs or a reproducible code repository; the numerical appendix describes experiments but no code or pretrained artifacts are included.","major_comments":[{"comment":"The network defined in Definition 2.4 does not receive time as an input. Equation (2) sets x^(0)_k = (s_{t_k}, a_k) and updates x^(l+1)_k = x^(l)_k + h_{θ_l}(x^(l)_k, a_k)Δt, with no t_k appearing in the recursion. For a fixed parameter vector θ, Q_θ(t,s,a) is therefore constant in t. Yet the target Q*(t,s,a) in Definition 2.13 depends on t through the remaining horizon; even in a degenerate MDP with h=σ=0, r=1, g=0 and a singleton action set, Q*(t,s,a) = (1-e^{-γ(T-t)})/γ is t-dependent. Theorem 3.1 asserts a uniform approximation over K_R = [0,T] x S_R x A in Eq. (21). This cannot hold for ε smaller than half the range of Q* in t. Appendix A.6 applies Lemma 2.8 to K_R, a compact subset of R^{1+n+m}, but Lemma 2.8 is stated for compact subsets of R^{n+m}; enlarging the ambient dimension does not repair the argument because the parametrized class of functions is a proper subclass of continuous functions, namely those independent of t. Moreover, Definition 2.4 only defines Q_θ at the grid times t_k, not for arbitrary (t,s,a), so the supremum in (21) is not even well-defined unless an interpolation is specified. Theorem 3.1 is therefore invalid as stated.","section":"Definition 2.4, Remark 2.5, and Appendix A.6"},{"comment":"The proof of Theorem 3.8 conflates the discrete-time Bellman fixed point with the continuous-time Q* of Definition 2.13. Equation (41) defines TQ = r + γ E[max_{a'} Q(t+Δt, s', a') | s,a], which is a discrete-time Bellman operator with discount γ per step. The continuous-time optimal Q-function, however, involves the discounted integral e^{-γ(u-t)} r du and the terminal term e^{-γ(T-t)} g(s_T); Eq. (13) itself only states that the discrete Bellman relation holds approximately as Δt -> 0. The claim 'the Bellman operator has a unique fixed point Q*' is not justified for this continuous-time Q*, and hence the convergence of θ_k to the parameter representing Q* does not follow from a contraction argument on the discrete operator. This is a load-bearing gap in the proof of the central convergence result.","section":"Appendix A.7, Eq. (41)"},{"comment":"Theorem 3.8 relies on assumptions that essentially postulate the conclusion. Assumption 3.5(i) assumes Q* is exactly representable in the network class, and Assumption 3.6(ii) assumes a negative-correlation inequality between the function error and the Bellman error. In the Lyapunov analysis of Appendix A.7, the proof computes the derivative of V(θ)=1/2||Q_θ-Q*||^2_{L^2(μ)}, writes 'Relating this directly to Assumption 3.6 (ii) requires further steps...', and then simply asserts that Assumption 3.6(ii) implies the dynamics drive θ to θ*. No derivation from the network structure, the Bellman operator, and the sampling measure is given. The theorem also claims convergence in the supremum norm over [0,T]×S×A, while the assumptions and Lyapunov function are formulated in a weighted L^2(μ) norm; the passage from θ_k -> θ* to ||Q_{θ_k}-Q*||_∞ -> 0 is not established, since continuity on a compact set plus large deviation bounds do not give uniform convergence on the whole, unbounded state space. As a result, Theorem 3.8 is not proven beyond restating the dissipativity condition.","section":"Assumptions 3.5-3.6 and Appendix A.7"}],"minor_comments":[{"comment":"The proof treats vector-valued approximation by approximating each component with a separate residual block and then stacking outputs. This is acceptable for a concatenated-output network, but it does not show that a single residual-block family of the form A_l η(B_l x + b_l) with a single activation layer can approximate an arbitrary vector-valued continuous function; that is a standard vector-valued UAT statement and should be stated and cited as such.","section":"Lemma 2.8 proof, Appendix A.3"},{"comment":"The proof asserts a specific exponential large-deviation bound with explicit constants C1, C2, C3, but it does not prove that bound or give a reference that applies to SDEs driven by general continuous square-integrable martingales. Since this lemma is used only to select R1, a weaker qualitative statement with a proper reference would be cleaner and avoids the current hand-constructed constants.","section":"Lemma 2.10 proof, Appendix A.4"},{"comment":"The remark suggests a rate L ∝ (1/ε)^κ (e.g., κ=2 for an Euler-Maruyama interpretation), but no rate theorem is proved or cited for the specific residual-block architecture of Definition 2.4. This is presented as speculation and should be explicitly marked as such.","section":"Remark 3.2"},{"comment":"The update rule in Eq. (22)-(23) omits the Δt scaling of the reward term that appears in the Section 2.4 target y_k = r Δt + e^{-γΔt} max Q; the convergence theorem uses the unscaled version, which is inconsistent with the continuous-time approximation described earlier.","section":"Theorem 3.8 statement"},{"comment":"There are numerous typos and spacing artifacts from the source text, such as 'timet', 'It ˆo', and broken reference formatting. These are cosmetic but should be cleaned before any resubmission.","section":"General presentation"}],"recommendation":"reject","confidential_remarks":"The paper's main approximation theorem is invalid for the architecture as defined because time is not an input, and the fix of adding a time coordinate would change the architecture and the experiments. The convergence theorem is also not proven, as the key Lyapunov condition is assumed rather than derived. These are load-bearing issues, not local presentation problems, so I recommend rejection. If the authors revise, they should address the time-input issue head-on and either prove the Lyapunov condition from the Bellman structure or substantially weaken Theorem 3.8."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things should know about arXiv:2505.02288. First, the approximation theorem fails as stated. Definition 2.4 builds the DQN from residual blocks whose input is (s_{t_k}, a_k); no coordinate t_k enters the recursion. So each Q_theta is constant in t. But Q* in Definition 2.13 genuinely depends on the remaining horizon, and in a trivial MDP Q*(t,s,a) = (1 - e^{-gamma(T-t)})/gamma, which varies with t. Theorem 3.1 promises uniform approximation on [0,T] x S x A, which cannot hold. Appendix A.6 applies Lemma 2.8 to a domain [0,T] x S x A, but Lemma 2.8 is for subsets of R^{n+m}; even extending it would not repair the argument because the parametrized function class is a proper subclass of continuous functions, namely those independent of t. This is not a minor typo; it is the paper's central claim.\n\nWhat the paper does well: it sets up a reasonable continuous-time MDP, correctly identifies that residual-network universal approximation plus continuity of Q* would give compact-set approximation, and it is honest about the strength of Assumptions 3.5 and 3.6. The FBSDE and viscosity-solution material is standard but not wrong. If time were added as an input and Q*'s continuity were proved from the coefficients rather than assumed, Theorem 3.1 would become a corollary of known results: useful as a bridge, not a new theorem.\n\nThe soft spots beyond the time issue: Theorem 3.8 assumes exact representability and a negative-correlation inequality, which essentially supply the conclusion. The proof in A.7 says 'requires further steps' and then proceeds conditionally. The discrete Bellman fixed point is never shown to equal the continuous-time Q* on the infinite domain, while the convergence is stated in the L-infinity norm over [0,T] x S x A. The experiments are not experiments: the text describes hypothetical figures and 'typical outcomes,' so there is no empirical support.\n\nWho this is for: a reader who wants a compact map of how one would try to extend stochastic-approximation and UAT arguments to continuous-time DQNs, and where the technical obstacles sit. That reader should not rely on the theorems as stated.\n\nRecommendation: desk reject, or major revision with time as an input, Q* continuity proved rather than assumed, a clear discrete-to-continuous fixed-point argument, and real experiments. As is, the central claim is not supported.","headline":"The main approximation theorem is invalid as stated because the DQN never sees time, and the convergence theorem is a conditional repackaging of known results.","tokens_in":24338,"tokens_out":3537,"would_cite":false,"duration_ms":45078,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","60H10","49L25","68Q32","93E35"],"pacs":[],"model":"deepseek-v4-flash","headline":"A residual-network DQN can match any continuous optimal Q-function on compact sets.","keywords":["deep Q-networks","universal approximation","continuous-time MDP","residual networks","Q-learning convergence","stochastic approximation","viscosity solutions","forward-backward SDE"],"falsifier":"Let $Q^*(t,s,a) = e^{t}g(s,a)$ for a nonzero Lipschitz $g$ on $[0,T] \\times S \\times A$. Any DQN whose input is $(s,a)$ only is a function independent of $t$, so its uniform distance from $Q^*$ is at least $(e^T - 1)/2$ times the maximum of $|g|$ over the compact region; a concrete check is whether any trained network achieves $\\sup |Q_\\theta - Q^*| < \\epsilon$ for $\\epsilon$ smaller than that bound, which would refute the approximation theorem as stated.","tokens_in":23243,"feed_emoji":"🎯","tokens_out":5740,"duration_ms":59520,"temperature":0.7,"pith_summary":"This paper tries to establish that Deep Q-Networks, built from residual blocks, are universal approximators for the optimal action-value function in a continuous-time Markov decision process driven by a square-integrable martingale. It claims that for any target accuracy and any high-probability threshold, a DQN with enough layers and parameters can match $Q^*$ uniformly on the compact region where the state-action trajectory actually lives. It further claims that the Q-learning rule used to train such a network converges to $Q^*$ almost surely, provided the network can represent $Q^*$, the parametrization is identifiable, and a negative-correlation condition links the Bellman error to the gradient. If correct, the result transfers classical universal approximation guarantees from residual networks to the reinforcement-learning setting and gives a stochastic-control foundation for DQN theory.","feed_headline":"A residual-network DQN can match any continuous optimal Q-function","feed_subtitle":"Residual networks give DQNs universal approximation on compact domains with high probability, and Q-learning can converge to the optimum.","key_machinery":"The argument runs on three objects. The DQN itself is a residual network whose hidden layers update $x^{(l+1)} = x^{(l)} + h_{\\theta_l}(x^{(l)}, a)\\Delta t$, the Euler discretization of an ODE, which lets the proof invoke residual-network universal approximation on compact sets. A large-deviation bound on the martingale-driven state process supplies the compact set $K_R$ carrying probability at least $1-\\delta$. For convergence, the Bellman operator $T(Q)(t,s,a) = r(t,s,a) + \\gamma\\mathbb{E}[\\max_{a'} Q(t+\\Delta t, s', a') \\mid s,a]$ is shown to be a $\\gamma$-contraction in the sup norm, so its fixed point is $Q^*$, and the averaged stochastic-approximation ODE $\\dot{\\theta} = \\bar{H}(\\theta)$ is steered to that fixed point by Lyapunov-style gradient conditions. Viscosity solutions of the HJB equation for $V^*$ play a supporting role in justifying the regularity of the value function without requiring classical differentiability of $Q^*$.","core_discovery":"On the paper's own terms, the central discovery is Theorem 3.1: under Lipschitz regularity on the MDP coefficients and continuity of $Q^*$, for every $\\epsilon > 0$ and every $\\delta \\in (0,1)$ there exists a residual-block DQN $Q_\\theta$ such that $\\sup_{K_R}|Q_\\theta(t,s,a) - Q^*(t,s,a)| < \\epsilon$, where $K_R = [0,T] \\times \\{s : \\|s\\| \\le R_1\\} \\times A$ is a compact set that the trajectory occupies with probability at least $1-\\delta$. Theorem 3.8 adds that the stochastic-approximation Q-learning iterates satisfy $\\|Q_{\\theta_k} - Q^*\\|_\\infty \\to 0$ almost surely under ergodic sampling, Robbins-Monro learning rates, representability, identifiability, and a gradient/Bellman-error correlation assumption. The two theorems together say that DQNs inherit the universal approximation capacity of ResNets for continuous $Q^*$, and that a natural training loop can in principle find the approximating parameters.","pith_inferences":["The paper leaves implicit that its network receives only $(s,a)$, not the time coordinate $t$; a fully honest statement of the approximation claim would require either feeding $t$ as an input or restricting to $Q^*$ with negligible time variation, and checking that gap is a natural next test.","One testable extension is to compare, on finite-horizon benchmark control tasks, the empirical approximation error of state-action-only DQNs versus networks with $t$ as an input; the paper's theorem predicts both can work in principle, while the gap would reveal how much time-dependence matters.","If combined with quantitative approximation rates for residual networks, the $L=N$ coupling suggests measurable scaling laws—depth as a function of $1/\\epsilon$ and of the modulus of continuity of $Q^*$—that could be validated on the 1D control experiment included in the paper.","The convergence proof's reliance on representability and the negative-correlation condition suggests that in practice, target networks and experience replay may be doing the work of enforcing these assumptions, and failures of convergence could be diagnosed as violations of Assumption 3.6 rather than of the Bellman contraction."],"forward_implications":["A DQN with enough residual layers and parameters can match any continuous optimal Q-function uniformly on the high-probability region of state-action space, so expressiveness is not the bottleneck in continuous-time DQN learning.","The depth choice $L=N$ ties network capacity to time discretization: refining the time grid naturally suggests deeper networks, and error $\\epsilon$ would require depth scaling roughly like $\\epsilon^{-\\kappa}$ under stronger smoothness assumptions.","Under representability, identifiable parametrization, and the negative-correlation gradient condition, Q-learning with target values converges to the true optimal Q-function almost surely, not merely to a stationary point.","The Bellman operator remains a contraction in the continuous-time discretized setting, so the classical Q-learning fixed-point logic survives the passage to martingale-driven SDEs.","Approximation error is controlled only on compact sets with high probability; the large-deviation radius $R_1$ absorbs the tail risk, so unbounded state spaces are handled probabilistically rather than uniformly."],"supporting_citations":[{"why":"Supplies the residual-network universal approximation result used to prove Lemma 2.8 and thus Theorem 3.1.","marker":"(Li et al., 2022)"},{"why":"Provides the classical multilayer-feedforward approximation theorem that the residual-block proof reduces to in Lemma 2.8.","marker":"(Hornik, 1991)"},{"why":"Sources the large-deviation principle that Lemma 2.10 converts into the high-probability compact set $K_R$.","marker":"(Dembo, 2009)"},{"why":"Cited for the large-deviation-type bound on SDEs driven by continuous martingales used in Lemma 2.10.","marker":"(Mao, 2007)"},{"why":"Supplies the stochastic-approximation convergence theorem that Theorem 3.8 adapts for the Q-learning iterates.","marker":"(Kushner & Yin, 2003)"},{"why":"Basis for the Bellman-operator contraction argument that identifies $Q^*$ as the unique fixed point in the convergence proof.","marker":"(Bertsekas & Tsitsiklis, 1995)"},{"why":"Provides the viscosity-solution and HJB background used to justify regularity and well-posedness of $V^*$ under Assumption 2.3.","marker":"(Fleming & Soner, 2006)"}],"fun_headline_variants":["Residual DQNs approximate any continuous Q-function","Universal approximation for Deep Q-Networks","DQNs match optimal Q-functions via residual nets","Deep Q-Networks go universal with residual blocks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central load-bearing premise is that a residual network whose inputs are only state and action can uniformly approximate $Q^*(t,s,a)$ on $[0,T] \\times S \\times A$ even though $Q^*$ depends on the remaining time horizon $t$; the paper's approximation proof applies a network approximation lemma stated for compact subsets of $\\mathbb{R}^{n+m}$ to the $(1+n+m)$-dimensional set $[0,T] \\times S \\times A$ without supplying $t$ as an input.","fun_headline_variants_meta":{"raw":{"variants":["Residual DQNs approximate any continuous Q-function","Universal approximation for Deep Q-Networks","DQNs match optimal Q-functions via residual nets","Deep Q-Networks go universal with residual blocks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0002,"raw_usage":{"total_tokens":1382,"prompt_tokens":958,"completion_tokens":424,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":574,"completion_tokens_details":{"reasoning_tokens":363}},"tokens_in":574,"tokens_out":424,"duration_ms":5194,"temperature":1.0,"reasoning_tokens":363,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:57:18.709466+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Let $Q^*(t,s,a) = e^{t}g(s,a)$ for a nonzero Lipschitz $g$ on $[0,T] \\times S \\times A$. Any DQN whose input is $(s,a)$ only is a function independent of $t$, so its uniform distance from $Q^*$ is at least $(e^T - 1)/2$ times the maximum of $|g|$ over the compact region; a concrete check is whether any trained network achieves $\\sup |Q_\\theta - Q^*| < \\epsilon$ for $\\epsilon$ smaller than that bound, which would refute the approximation theorem as stated.","supporting_citations":[],"review_version":1}