{"id":"a80d732d-dc30-4e9e-aeaa-c74c6dc0221f","arxiv_id":"2505.06023","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"A conditional universal approximation theorem shows that residual operator-block DQNs with depth aligned to Bellman iterations can approximate the optimal Q-function, assuming a neural operator class with controlled Lipschitz outputs exists.","lead":"The paper proves a universal approximation theorem for a class of residual deep Q-networks whose layers mimic Bellman value iteration, tying network depth directly to the number of refinement steps. It uses backward stochastic differential equations and Lipschitz regularity arguments to justify why the approximator class can represent the optimal Q-function.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The main theorem is conditional on Assumption 4.1, and the proof's construction of Ktarget and L*_unif-Lip is circular: L*_F is fixed for a set whose Lipschitz bound is defined through L*_F, with no fixed-point argument; the gap is demonstrable even in a scalar MDP.","rationale":"The reader's weakest assumption correctly identifies Assumption 4.1 and the circular dependency between L*_F and Ktarget as the critical gap. My closer analysis of Appendix C confirms that this is not merely a missing technical detail: the proof literally defines the compactness set by a Lipschitz bound that depends on the output Lipschitz constant of the neural operator, which is then selected after that set is fixed. Remark 4.2 tries to break the circle by declaring L*_F a 'given constant,' but Assumption 4.1(b) allows L*_F to depend on the task's Lipschitz data, and a lower-bound argument shows that any epsilon-accurate approximant of J on a set of L-Lipschitz functions must itself have output Lipschitz constant proportional to L. Plugging that dependency back into the Appendix C recurrence produces an inequality that cannot be satisfied for the layer counts L required by the contraction argument, at least on a simple scalar MDP. This makes the proof's central induction invalid. I do not claim the theorem is false; it may be salvageable with a different architecture or a non-circular finite-horizon construction, but the paper as written does not supply such a proof. I also note the separate circularity in Lemma 3.1(d), where Lemma 3.2 is invoked to justify uniform Lipschitz continuity while Lemma 3.2 relies on that same uniform Lipschitz continuity; the Assumption 4.1 gap is already sufficient for rejection, so I have focused the attack there. The reader's verdict of REJECT with moderate confidence is therefore appropriate.","tokens_in":24844,"tokens_out":12799,"duration_ms":141318,"concrete_test":"Instantiate the argument on the scalar zero-dynamics MDP with BQ = gamma Q + r, r(s)=s, gamma = e^{-lambda delta}=0.9, so J(Q)=r-(1-gamma)Q. Let K_L be the set of all L-Lipschitz functions on [0,1] bounded by 1. (1) Show that any operator achieving sup-norm error <= eps over K_L must have output Lipschitz constant L*_F >= (1-gamma)L + K_A - O(eps/diam), by comparing two nearby points where J(Q) attains its maximal slope. (2) Substitute this lower bound into the Appendix C recurrence L_{l+1} <= (2K_A + L*_F) + (2K_B + 1)L_l, with K_A = (1-gamma)/lambda, K_B = e^{-lambda delta}(1+CS(delta)) as in Appendix A. (3) For gamma=0.9 and L=3 layers, compute the resulting bound on L*_unif-Lip = (2K_A + L*_F)(1 + B' + (B')^2), with B'=2K_B+1.","verdict_should_be":"REJECT","load_bearing_attack":"The load-bearing gap is in the proof of Theorem 4.4, Appendix C, Step 3, together with Assumption 4.1(b) and Remark 4.2. The compact set Ktarget is defined as all functions that are M*_unif-bound-bounded and L*_unif-Lip-Lipschitz. But L*_unif-Lip is itself defined by the recurrence L_{l+1} <= (2K_A + L*_F) + (2K_B + 1)L_l, so L*_unif-Lip = A'(L*_F) * sum_{j=0}^{L-1} (B')^j. Lemma 4.3 then invokes Assumption 4.1(b) to supply L*_F for an architecture that approximates J on exactly this Ktarget. Assumption 4.1(b) explicitly permits L*_F to depend on the Lipschitz data of the task, i.e. on LK = L*_unif-Lip, and any realistic realization has L*_F growing with L*_unif-Lip: the output function must track J(Q), whose Lipschitz constant is at least K_A + (1 - e^{-lambda delta})L for Q in K_L, so L*_F cannot be bounded independently of L*_unif-Lip. Remark 4.2 does not resolve this; it only asserts that L*_F is a 'given constant' for the eventual Ktarget, which is precisely the circular step. Substituting the unavoidable lower bound L*_F >= K_A + (1 - e^{-lambda delta})L*_unif-Lip into the Appendix C recurrence yields an inequality with no positive solution once L is moderately large, so the induction proving that all approximating iterates lie in Ktarget cannot be closed. Standard neural operator UATs only provide sup-norm approximation on a prescribed compact set; they do not simultaneously control output-function Lipschitz constants on a set whose defining Lipschitz constant is defined through those same output constants. The central theorem is therefore conditional on an unproven existence claim that is close in strength to the result being proved.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a DQN architecture composed of neural-operator residual blocks, where each block approximates the residual Bellman operator J(Q)=BQ-Q, and claims a universal approximation theorem for the resulting network. The proof strategy is to show that the Bellman iterates Q(k) are uniformly bounded and uniformly Lipschitz, to place them in a compact subset of C(KQ), to apply a neural-operator universal approximation theorem for the operator J on that compact set, and to control the accumulation of per-layer errors using the contraction of the Bellman operator. The main theorem, Theorem 4.4, states that for any epsilon>0 there are L layers, a discretization scheme, and parameters such that the final network output approximates Q* to within epsilon in the sup norm, with L related to T/delta and the contraction factor e^{-lambda delta}.","tokens_in":25293,"tokens_out":6045,"duration_ms":59742,"significance":"If the claims were correct, the paper would offer a structural, problem-aware UAT for DQNs and a transparent link between network depth and Bellman iterations, with a clean error-propagation analysis. The operator view of residual layers and the explicit accumulation bound e_L <= epsilon_1/(1-e^{-lambda delta}) in Appendix C are useful and clearly presented. However, the result does not currently hold as proved: the uniform Lipschitz lemma is circular, and the central Assumption 4.1 is both unproved and used in a circle through the definition of K_target. These are not presentation issues but load-bearing gaps in the proof of Theorem 4.4.","major_comments":[{"comment":"The argument proving uniform Lipschitz continuity of all iterates is circular. After deriving the recurrence L(k+1) <= K_A + K_B L(k), the proof considers the case K_B >= 1 and asserts that because Q(k) -> Q* uniformly and Q* is Lipschitz, the set {Q(k)} union {Q*} is compact by Lemma 3.2, which in turn is proved from Lemma 3.1(d). The additional appeal to 'the number of relevant recursions is bounded by N_max = ceil(T/delta)' is not justified: the value iteration sequence is indexed by k >= 0 and the recurrence has no built-in truncation at N_max; if K_B >= 1, the displayed bound involving N_max applies only to the first N_max iterates and says nothing about later iterates unless one already knows uniform Lipschitz boundedness. Thus the uniform regularity of all iterates, which is the load-bearing premise for Lemma 3.2 and for the compactness needed by the neural-operator UAT, is not established.","section":"Appendix A, proof of Lemma 3.1(d)"},{"comment":"The definition of the compact set K_target is circular. In Step 3 of Appendix C, L*_unif-Lip is defined through the quantities A' = 2K_A + L*_F and B' = 2K_B + 1, so L*_unif-Lip depends on L*_F. Lemma 4.3 then invokes Assumption 4.1(b) to obtain an L*_F for an architecture approximating J on exactly this set K_target = {Q : ||Q||_infty <= M*_unif-bound, Lip(Q) <= L*_unif-Lip}. Assumption 4.1(b) explicitly allows L*_F to depend on the Lipschitz data L_K of the task, and since J(Q) = BQ - Q has Lipschitz constant at least K_A + (1 - e^{-lambda delta}) L for Q in K_target, any realization has L*_F >= K_A + (1 - e^{-lambda delta}) L*_unif-Lip; substituting this into the recurrence gives no positive solution for L*_unif-Lip once L is moderately large, so the induction that all approximate iterates lie in K_target cannot be closed. Remark 4.2 only asserts that L*_F is a 'given constant' for the eventual K_target; that is precisely the fixed-point step that is missing.","section":"Assumption 4.1 and Appendix C, Step 3"},{"comment":"The main theorem is conditional on an unproved existence statement. Assumption 4.1(b) postulates that, for every compact set of uniformly Lipschitz functions, a neural operator can uniformly approximate a given continuous operator while all output functions share a uniform Lipschitz constant L*_F. Standard neural-operator UATs cited in the paper provide sup-norm approximation on a prescribed compact set, but they do not control the Lipschitz constant of the output functions. Remark 4.2's basis-function estimate L*_F <= M B_y L_phi is only a sketch: it assumes bounded network outputs B_y and does not show that those bounds are compatible with the required approximation accuracy epsilon_op for J on K_target, nor does it constitute a proof of existence for the architecture used in Lemma 4.3. Since Theorem 4.4 is stated only under this assumption, the paper's central approximation claim is conditional on a property whose realization is not demonstrated.","section":"Assumption 4.1(b)"}],"minor_comments":[{"comment":"The sentence 'we set L = 1 if epsilon >= 2M_Q' is inconsistent with the preceding formula L = ceil(ln(2M_Q/epsilon)/(lambda delta)) + 1, since the formula already handles all epsilon > 0.","section":"Appendix C, Step 1"},{"comment":"The continuity argument relies on a convergence statement for SDE solutions on a common time interval but does not spell out how the integrals with different lower limits t and t' are compared; this is a presentation gap in a non-central part of the proof.","section":"Appendix A, proof of Lemma 3.1(a)"},{"comment":"The axis tick labels in Figure 1 are rendered as raw Unicode codepoints, making the figure unreadable; the figure should be regenerated.","section":"Figure 1"},{"comment":"Lemma 4.3 is stated before the set K_target is defined and refers to 'as defined in the proof of Theorem 4.4 in Appendix C and shown therein', which makes the lemma difficult to read independently; a self-contained definition before the lemma would improve clarity.","section":"Lemma 4.3 and Appendix C"},{"comment":"The citation 'Krylov and Krylov [1987]' appears in the text but the reference list contains a single Krylov entry with the same author name repeated; this should be corrected.","section":"Section 5.1.1"}],"recommendation":"reject","confidential_remarks":"The manuscript would require a genuinely new regularity argument for Lemma 3.1(d) and a construction of a neural operator class satisfying Assumption 4.1(b) with a non-circular dependence on the target set. These are substantial changes, so I recommend rejection despite the interesting framing of depth as Bellman iteration."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper has one idea that deserves attention: it builds a DQN out of neural operator blocks that each approximate the Bellman residual J(Q) = BQ − Q, and then argues that network depth plays the role of value-iteration count. If the main theorem (4.4) were airtight, it would give a UAT tuned to the MDP's structure, with the Lipschitz regularity of the Bellman iterates doing real work. That is a genuine conceptual advance over applying a generic UAT to Q*.\n\nThe paper also does some things well. The setup of the δ-discretized Bellman equation and the BSDE motivation are clearly written. The error propagation in Section 5, where the per-layer operator error accumulates via the contraction factor, is the right intuition and is presented cleanly. And the author is transparent that the hard part is the Lipschitz control of the operator outputs (Assumption 4.1).\n\nBut the proof as written does not close. The regularity lemma, Lemma 3.1(d), is proved by a recurrence L(k+1) ≤ K_A + K_B L(k) and then, when K_B ≥ 1, the proof appeals to compactness via Lemma 3.2 to get uniform boundedness of the L(k). Lemma 3.2 is itself derived from uniform Lipschitz continuity. That is circular. The finite-horizon remark points in a plausible direction, but the finite-stage induction that would rescue it is not actually given.\n\nThe bigger problem is Assumption 4.1. The main theorem leans on the existence of a neural operator class that simultaneously gives sup-norm approximation on K_target and a uniform output-Lipschitz bound L*_F. But L*_F enters the recurrence that defines L*_unif-Lip, which in turn defines K_target. Remark 4.2 asserts that L*_F can be treated as fixed, but it does not show that a fixed point exists. The stress-test argument shows that for realistic constructions L*_F must grow with L*_unif-Lip, and then the induction cannot be closed. So Theorem 4.4 is conditional on an assumption that is close in strength to the result.\n\nThis is not a paper to desk-reject, because the framing is novel and the gaps are explicit enough to give a referee something to work with. It is also not a paper to accept. The path forward is clear: prove the uniform Lipschitz lemma by a direct finite-horizon induction, and either construct the operator class in Assumption 4.1 or replace it with a weaker stability condition.\n\nYou could bring it to a reading group to discuss what a valid UAT for DQNs needs. I would cite the conceptual framing. Send it to peer review with the expectation of heavy revision.","headline":"A new and appealing architecture-to-Bellman-iteration correspondence, but the main theorem leans on a circular regularity proof and an operator existence assumption that is close to the result itself.","tokens_in":25855,"tokens_out":6841,"would_cite":true,"duration_ms":64874,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"For any target accuracy, a residual Q-network whose layers emulate Bellman updates can match the optimal Q-function to that accuracy, with depth set by the horizon and the discount.","keywords":["universal approximation theorem","deep Q-networks","Bellman operator","value iteration","neural operators","backward stochastic differential equations","Lipschitz regularity","residual networks"],"falsifier":"Take a simple MDP satisfying Assumption 2.1 (for instance, the one-dimensional deterministic action problem in Appendix E) and compute, as $\\varepsilon \\to 0$, the minimal output Lipschitz constant $L^*_F$ among all neural-operator blocks that approximate $J$ on the compact set of iterates to sup-norm accuracy $\\varepsilon$. If $L^*_F$ cannot be bounded independently of $\\varepsilon$, Assumption 4.1(b) fails and the theorem's conclusion is not guaranteed.","tokens_in":24531,"feed_emoji":"🤖","tokens_out":14698,"duration_ms":127057,"temperature":0.7,"pith_summary":"Standard universal approximation theorems justify deep Q-networks by treating the optimal Q-function as an arbitrary continuous function. This paper argues that $Q^*$ is not arbitrary: it is the unique fixed point of a Bellman operator with a contraction structure, and a network built from operator layers that approximate the Bellman residual can exploit that structure. The central claim is a universal approximation theorem for such networks: for any $\\varepsilon > 0$, enough layers and a fine enough discretization grid give a network output within $\\varepsilon$ of $Q^*$ in the sup norm. Depth plays a concrete role, corresponding to Bellman iterations, and the discount factor controls how operator approximation errors accumulate. A sympathetic reader should care because this gives a problem-aware, structural explanation of DQN approximation power rather than a generic function-approximation guarantee.","feed_headline":"Proof: layers of a deep Q-network can emulate Bellman updates","feed_subtitle":"If right, each network layer performs one value-iteration step and depth is fixed by the discount and horizon.","key_machinery":"The load-bearing object is the Bellman operator $B$ on the space $C(K_Q)$ of continuous Q-functions, defined for a $\\delta$-discretized continuous-time MDP in which the chosen action is held constant for a short duration $\\delta$. The operator $B$ is a contraction with factor $e^{-\\lambda\\delta}$, and its residual form $J(Q) = BQ - Q$ is what each network block approximates. A single application of $B$ can be represented as the solution of a short-horizon backward stochastic differential equation coupled to the forward state process, which is the FBSDE system of the title; this representation is what transfers regularity from the input $Q$ to the output $BQ$. The network is a residual stack of neural-operator blocks of the form $Q \\mapsto Q + D_M(N_\\theta(E_M(Q)))$, where $E_M$ samples $Q$ on a grid, $N_\\theta$ is a finite-dimensional network, and $D_M$ reconstructs a function. The proof's key step, Lemma 3.1(d), establishes that all Bellman iterates are uniformly Lipschitz and uniformly bounded, placing them in a compact subset of $C(K_Q)$ so that the neural-operator approximation lemma applies uniformly across layers.","core_discovery":"The paper's central claim is Theorem 4.4: for any $\\varepsilon > 0$, there exist a number $L$ of operator layers, a discretization scheme $(E_M, D_M)$, and network parameters $\\theta$ such that the network's final output satisfies $\\|\\hat{Q}_{NN}^{(L)} - Q^*\\|_\\infty < \\varepsilon$, where $L$ is tied to $T/\\delta$ and the contraction factor $e^{-\\lambda\\delta}$. The proof shows that, under the standard Lipschitz assumptions on the MDP data, every Bellman iterate $Q^{(k)}$ and the limit $Q^*$ is uniformly Lipschitz and uniformly bounded on the compact domain $K_Q$. These functions therefore lie in a compact set of continuous functions, which allows a neural-operator universal approximation theorem to approximate the residual operator $J(Q) = BQ - Q$ uniformly on that set. Each residual network block then acts as one approximate Bellman step, and the contraction of $B$ keeps the accumulated approximation error from exploding. The paper presents this as an iterative-refinement proof technique in which network depth literally corresponds to value-iteration count.","pith_inferences":["A natural next step, not taken in the paper, is to instantiate Assumption 4.1 for concrete architectures such as bounded-basis decoders, which would turn the conditional theorem into an unconditional one.","The FBSDE perspective suggests that structure-aware universal approximation theorems could be built for other control problems whose value functions have known regularity classes; this paper's construction is one instance of that pattern.","A direct empirical test of the key assumption would be to train the proposed operator-residual network on a small continuous-time MDP at several decreasing target errors and measure the Lipschitz constant of the learned operator blocks; boundedness of that constant would support Assumption 4.1, while divergence would indicate the assumption needs modification."],"forward_implications":["Depth is assigned a quantitative meaning: a network with $L \\approx (1/(\\lambda\\delta)) \\log(M_Q/\\varepsilon)$ layers performs roughly $L$ approximate Bellman steps, and adding layers is equivalent to running more value iterations.","The contraction factor $e^{-\\lambda\\delta}$ guarantees that per-layer operator approximation errors accumulate at most as $1/(1-e^{-\\lambda\\delta})$; the total error is the value-iteration truncation error plus this accumulated operator error.","Under the theorem's assumptions, the optimal Q-function and every Bellman iterate lie in a common compact, uniformly Lipschitz set, which is exactly the setting where neural-operator UATs apply; this justifies the use of operator-based residual architectures for Q-learning.","Quantitative rates require stronger smoothness of the Bellman iterates: with $s$-order smoothness, the discretization error scales like $M^{-s/d_Q}$, so improving regularity estimates on $Q^*$ directly improves the grid size and network size needed for a given $\\varepsilon$."],"supporting_citations":[{"why":"Supplies the universal approximation theorem for neural operators that Assumption 4.1 and Lemma 4.3 rely on.","marker":"Kovachki et al. [2023]"},{"why":"Earlier operator universal approximation result used as the foundation for the decoder-network-encoder block construction.","marker":"Chen and Chen [1995]"},{"why":"Classical universal approximation theorem that the paper contrasts; it approximates continuous functions without exploiting Bellman structure.","marker":"Cybenko [1989]"},{"why":"Introduces DQN, the architecture whose approximation power this paper sets out to justify.","marker":"Mnih et al. [2015]"},{"why":"Provides the BSDE existence, uniqueness, and regularity theory used to analyze a single Bellman step.","marker":"El Karoui et al. [1997]"},{"why":"Supplies SDE stability estimates with respect to initial data, used in the Lipschitz propagation proof of Lemma 3.1(d).","marker":"Yong and Zhou [1999]"},{"why":"Gives viscosity-solution and value-function regularity results used to conclude that Q* and its iterates are Lipschitz.","marker":"Fleming and Soner [2006]"},{"why":"Used for Lipschitz regularity of the BSDE solution Y_t with respect to initial data, which transfers regularity to BQ.","marker":"Pardoux [1999]"}],"fun_headline_variants":["Deep Q-networks emulate Bellman updates layer by layer","Each layer of a deep Q-network performs a Bellman step","Network depth equals value-iteration count in Q-learning","Bellman operator approximated by residual network layers","Universal approximation via Bellman-refining network depth"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The theorem rests on Assumption 4.1: that there exists a family of neural-operator blocks which can approximate the Bellman residual $J(Q)$ to any desired accuracy while keeping the Lipschitz constant of their output functions uniformly bounded; the paper sketches why bounded basis functions might achieve this but does not prove that such a family exists.","fun_headline_variants_meta":{"raw":{"variants":["Deep Q-networks emulate Bellman updates layer by layer","Each layer of a deep Q-network performs a Bellman step","Network depth equals value-iteration count in Q-learning","Bellman operator approximated by residual network layers","Universal approximation via Bellman-refining network depth"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000194,"raw_usage":{"total_tokens":1383,"prompt_tokens":1006,"completion_tokens":377,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":622,"completion_tokens_details":{"reasoning_tokens":301}},"tokens_in":622,"tokens_out":377,"duration_ms":4089,"temperature":1.0,"reasoning_tokens":301,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:51:04.944433+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a simple MDP satisfying Assumption 2.1 (for instance, the one-dimensional deterministic action problem in Appendix E) and compute, as $\\varepsilon \\to 0$, the minimal output Lipschitz constant $L^*_F$ among all neural-operator blocks that approximate $J$ on the compact set of iterates to sup-norm accuracy $\\varepsilon$. If $L^*_F$ cannot be bounded independently of $\\varepsilon$, Assumption 4.1(b) fails and the theorem's conclusion is not guaranteed.","supporting_citations":[{"cited_title":"Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and its application to dynamical systems","cited_arxiv_id":null,"evidence_quote":"Earlier operator universal approximation result used as the foundation for the decoder-network-encoder block construction."},{"cited_title":"Human-level control through deep reinforcement learning","cited_arxiv_id":null,"evidence_quote":"Introduces DQN, the architecture whose approximation power this paper sets out to justify."},{"cited_title":"Backward stochastic differential equations in finance","cited_arxiv_id":null,"evidence_quote":"Provides the BSDE existence, uniqueness, and regularity theory used to analyze a single Bellman step."},{"cited_title":"Controlled Markov processes and viscosity solutions, volume 25","cited_arxiv_id":null,"evidence_quote":"Gives viscosity-solution and value-function regularity results used to conclude that Q* and its iterates are Lipschitz."},{"cited_title":"Bsdes, weak convergence and homogenization of semilinear pdes","cited_arxiv_id":null,"evidence_quote":"Used for Lipschitz regularity of the BSDE solution Y_t with respect to initial data, which transfers regularity to BQ."}],"review_version":1}