{"id":"f35d3193-ec5c-4796-a0f6-8b0c38d91cef","arxiv_id":"2507.15082","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A new robust control PDE with gradient uncertainty is introduced, the quadratic LQ ansatz is shown to fail, and a GURAC algorithm is proposed, but with incomplete proofs and missing artifacts.","lead":"This paper proposes a new robust control framework where an adversary can also distort the gradient of the value function, not just the system dynamics. It derives a new nonlinear PDE and an actor-critic algorithm, but key theoretical proofs contain gaps and the empirical study is not reproducible from the paper alone.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The zero-sum game behind GU-HJBI is never defined and no dynamic programming principle is proved; Eq. (10) is an asserted PDE, so the paper's central game interpretation and downstream penalties are unsupported.","rationale":"The reader's weakest-assumption analysis identifies the same structural gap: Eq. (10) is asserted as the dynamic programming form of a zero-sum game without defining the game or proving a DPP. I agree that this is the single most load-bearing concern, because the game interpretation is what distinguishes GU-HJBI from an arbitrary nonlinear PDE and because every later contribution (perturbation expansion, numerical validation, GURAC) is derived from Eq. (10). The paper's own text supports the concern: Section 4.3 contains the unsupported sentence that the value function of 'the underlying game' is the unique solution, and no admissible strategy sets or payoff are given anywhere. Two further weaknesses reinforce the rejection but are secondary. First, the proof of Proposition 5.2 in Section 5.1 substitutes the control that minimizes only the quadratic part of the Hamiltonian, ignoring the epsilon-norm term; the infimum over u of a quadratic plus a non-smooth term is not attained at that control, so the contradiction proves nothing about the exact equation. Second, Step 4 of the comparison-principle proof in Appendix C claims that the difference of the gradient-uncertainty Hamiltonians vanishes because p_x - p_y -> 0 and the functions are uniformly continuous; this forgets that p_x = alpha(x-y) + beta x can itself be unbounded, and the Hamiltonian is quadratic in p, so continuity in p alone does not justify the limit. These do not change the verdict: REJECT remains appropriate, and the central idea could become viable only with a rigorous game definition, a DPP proof, and a corrected LQ argument.","tokens_in":22722,"tokens_out":17903,"duration_ms":225916,"concrete_test":"Specify a concrete game for the scalar LQ example of Section 6.1: state X_{t+1} = X_t + (aX_t + b u_t + sigma h_t) dt + sqrt(dt) sigma xi_t, with adversary controls h_t and delta_t, an explicit information structure (e.g., both observed before the Brownian increment), and payoff sum_t e^{-rho t}[L(x_t,u_t) dt - (1/(2 eta)) h_t^2 dt + delta_t (aX_t + b u_t + sigma h_t) dt]. Compute the finite-horizon value by backward induction on a fine grid and compare with the stationary viscosity solution of Eq. (11). If the two disagree beyond discretization error, the asserted DPP is false; if no such game can be specified because delta is intended to perturb grad V, then Eq. (10) has no game-theoretic derivation as written.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3 introduces Eq. (10) as the 'dynamic programming form' of a zero-sum game in which the adversary perturbs both the drift h and the value gradient delta, but the paper never specifies the game: no payoff functional, no admissible strategy spaces, and no information structure for delta (state-dependent? adapted? full knowledge of grad V?). No dynamic programming principle is stated or proved. This is not a cosmetic omission: in the standard robust-control game (5)-(6), Eq. (8) is a theorem for a well-defined payoff, whereas here delta enters the Hamiltonian through p+delta = grad V + delta, so V itself appears inside the supremum. A standard differential-game DPP would place delta in the running cost, and then delta would not have the claimed 'gradient uncertainty' interpretation. Theorem 4.5 further asserts 'It can also be shown that the value function defined by the underlying game is this unique solution' without ever defining that game. Since the small-epsilon expansion Eq. (13), the perturbation analysis of Section 5, the numerical studies of Section 6, and the GURAC loss of Section 8 all inherit this identification, the central claim that GU-HJBI is the correct dynamic programming equation for the proposed problem is unestablished.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a robust control framework in which an adversary perturbs both the system dynamics and the gradient of the value function, leading to a new PDE called the Hamilton-Jacobi-Bellman-Isaacs Equation with Gradient Uncertainty (GU-HJBI). After deriving this equation heuristically, the paper claims well-posedness via viscosity-comparison and Perron-type existence theorems, proves that the quadratic value-function ansatz fails in the linear-quadratic case, gives a formal perturbation expansion and numerical solutions, and finally proposes a GURAC-TD3 reinforcement-learning algorithm with an empirical study on Pendulum-v1.","tokens_in":23016,"tokens_out":6431,"duration_ms":71526,"significance":"The problem is well motivated, and the GU-HJBI equation is an interesting object: if the game-theoretic identification, well-posedness, and LQ non-quadraticity were rigorously established, the paper would be a meaningful extension of robust control and a useful conceptual bridge to RL. The manuscript is self-contained in notation and includes concrete numerical studies (Section 6) and an empirical comparison of GURAC-TD3 against TD3 (Section 8), which are strengths; the derivations are also largely self-contained rather than circular. However, the load-bearing derivation of the dynamic programming equation is asserted rather than proved, the LQ proof applies only to a substituted control in an approximate equation, and the well-posedness theorems do not cover the LQ setting used for the main structural claim. The contribution is therefore presently suggestive rather than established.","major_comments":[{"comment":"The central PDE is introduced as the 'dynamic programming form' of a zero-sum game in which the adversary chooses both h and δ, but no game is actually defined: there is no payoff functional, no admissible strategy space for δ (state-dependent? adapted? with full knowledge of ∇V?), and no dynamic programming principle is stated or proved. Because the gradient perturbation enters the Hamiltonian as p+δ = ∇V(x)+δ, V itself appears inside the supremum, so the standard differential-game DPP used for Eqs. (5)-(8) cannot be invoked by analogy. Theorem 4.5 then asserts, without derivation, that 'the value function defined by the underlying game is this unique solution.' This is the load-bearing link between the game interpretation and all downstream results (Eq. (13), Section 5, Section 8), and it remains unsupported.","section":"Section 3, Eq. (10)"},{"comment":"The proof substitutes u*(x) = -R^{-1}B^T P x, which minimizes only the quadratic part of the objective, into the approximate equation, and observes that the result contains ϵ||Mx||. This does not establish the proposition, because the infimum in Eq. (11) ranges over all controls; a nonlinear control could in principle compensate the norm term, and the proof gives no argument that it cannot. Moreover, Eq. (15) is the first-order approximation (13), not the full GU-HJBI equation (11), so the claim that the quadratic ansatz fails for the full equation for any ϵ>0 is not proved.","section":"Section 5.1, Proposition 5.2 and Eq. (15)"},{"comment":"The comparison-principle proof is only a sketch: the key continuity step G(x,u,px)-G(y,u,py)→0 is asserted rather than proved, and the treatment of the semifinal jet estimates is abbreviated. More importantly, Theorem 4.5 requires bounded viscosity sub-/supersolutions, while Assumption 2.1's growth condition on L is violated by the quadratic LQ cost x^T Qx used in Section 5; constant barrier functions cannot control an unbounded running cost on all of R^n. In addition, Section 5 sets U = R^k, whereas Assumption 2.1 requires a compact control set, so the viscosity results of Section 4 are not formally applicable to the LQ case on which the main structural claim rests.","section":"Section 4, Theorems 4.3 and 4.5; Appendix C"},{"comment":"The perturbation expansion is formal. The source term ||Aeff,0 x|| is not differentiable at x = 0, so the Implicit Function Theorem justification in Appendix G cannot hold in the C^2 or Hölder spaces where the linearized operator Llin is elliptic; likewise u1(x) = -R^{-1}B^T∇V1(x) requires differentiability of V1, which has not been established for a PDE with a non-smooth source. Consequently, the conclusion that the optimal control law is nonlinear, and the numerical solutions of Eq. (20), should be regarded as heuristic rather than proven.","section":"Section 5.2, Eq. (18) and Appendix G"}],"minor_comments":[{"comment":"The high-frequency oscillations in the reported control law are attributed to numerical artifacts; a convergence study under mesh refinement would materially strengthen the claim that the nonlinear character is genuine rather than an artifact.","section":"Section 6, Figure 1"},{"comment":"The GURAC penalty requires estimates of the environment dynamics f and diffusion σ, and the paper states that these are 'assumed known' for Pendulum-v1; this is a strong modeling assumption that should be stated as a limitation of the method, not as an implementation detail.","section":"Section 8, Algorithm 1"},{"comment":"There are repeated notation slips, e.g., 'D2V V' in Appendix E and in the definition of the generator in Section 2.1; these should be corrected to D^2 V or to a single consistent symbol for the Hessian argument.","section":"Throughout"},{"comment":"No code, data, or hyperparameter ranges are provided for the GURAC experiments, so the empirical claims in Section 8.4 cannot be independently reproduced or checked for statistical sensitivity.","section":"Throughout"}],"recommendation":"reject","confidential_remarks":"The missing game definition and the unproved dynamic programming principle are decisive. The paper's strongest result, the LQ non-quadraticity claim, is proven only against a particular substituted control in an approximate equation, and the well-posedness section does not apply to the LQ setting because of the noncompact control set and unbounded cost. These are not merely presentation issues; they affect the central claims. I would not encourage a quick revision unless the authors can provide a rigorous derivation of the dynamic programming equation for a properly defined game."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Qi's paper has a genuinely novel idea: treat the agent's uncertainty about the value-function gradient as an adversary that perturbs the gradient inside the HJBI Hamiltonian. That produces a new PDE (GU-HJBI) and a plausible story for why function-approximation RL is unstable. The paper also does some things well: the comparison principle under uniform ellipticity is a standard adaptation but carefully stated, the numerical solutions of the linearized first-order PDE look consistent, and the GURAC algorithm is a concrete, sensible translation of the idea to actor-critic.\n\nThe problems are all in the load-bearing layer. First, the paper never defines the zero-sum game it claims to solve. Eq. (10) is asserted as the \"dynamic programming form\" but there is no payoff functional, no admissible strategy spaces, no information structure for the gradient perturbation, and no proof of a dynamic programming principle. Theorem 4.5 even refers to \"the value function defined by the underlying game\" when no game has been defined. This is not cosmetic: with the gradient perturbation entering as a shift of the value gradient inside the supremum, the equation is not a standard differential-game DPP, and the interpretation of the perturbation as gradient uncertainty depends on a game that isn't specified. Second, the proof of Proposition 5.2 substitutes the control that minimizes only the quadratic part of the objective; that is not the infimizing control, so the equality in (15) is an inequality, and the contradiction doesn't follow. The claim may be true, but the proof as written is wrong. Third, the numerical studies solve the approximate equation (13), not the full GU-HJBI, so they validate the perturbation expansion, not the central PDE. The RL experiments are described in enough detail to be plausible but not enough to be reproduced without code.\n\nSo: interesting idea, honest exposition, but the formal core isn't established. The missing DPP and the flawed LQ proof are fixable, but they are central. I'd send it to review because the idea deserves careful referee time, but I'd expect major revision or rejection in current form. If the author can define a proper game and prove a DPP (or state the PDE as the primary object and demote the game interpretation), and fix the LQ proof, the core could be salvageable.","headline":"A genuinely new formulation of robust control that targets gradient uncertainty, but the central game is never defined and the key LQ proof is flawed; the idea is worth a serious referee, not acceptance as-is.","tokens_in":23468,"tokens_out":2849,"would_cite":false,"duration_ms":30309,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["49L25","49N70","93E20"],"pacs":[],"model":"deepseek-v4-flash","headline":"A robust-control framework treats the value function gradient itself as an adversarial target, yielding a well-posed PDE and showing that any gradient uncertainty destroys the quadratic structure of linear-quadratic control.","keywords":["robust control","gradient uncertainty","Hamilton-Jacobi-Bellman-Isaacs equation","viscosity solutions","linear-quadratic control","perturbation analysis","actor-critic reinforcement learning","zero-sum dynamic games"],"falsifier":"Compute, on a fine grid, the viscosity solution of (11) for the scalar LQ problem with $\\epsilon>0$ and compare it with the dynamic-programming value of an explicitly discretized version of the proposed zero-sum game; any discrepancy would show that equation (10) is not the game's dynamic programming equation. As a lighter check, test Proposition 5.2 by evaluating the second derivative of the numerically computed value function: a constant Hessian away from the origin would refute the claimed non-quadratic structure.","tokens_in":22499,"feed_emoji":"🎛️","tokens_out":11518,"duration_ms":109394,"temperature":0.7,"pith_summary":"The paper asks how a controller should act when it is uncertain not only about the model driving its system, but also about the marginal value of its own state. It answers by adding a second adversarial perturbation, chosen pointwise from a ball of radius $\\epsilon$, to the gradient of the value function inside a standard robust zero-sum game. That leads to a new dynamic programming equation, the GU-HJBI equation, whose viscosity solutions are shown to exist and be unique under uniform ellipticity. In the linear-quadratic case the paper proves the classical quadratic value function ansatz fails for any $\\epsilon>0$, and a perturbation expansion shows the correction is non-polynomial, making the optimal control law nonlinear. The paper then converts the theory into a practical regularizer for actor-critic reinforcement learning and reports that it stabilizes training on a pendulum task.","feed_headline":"Gradient uncertainty kills the quadratic value function","feed_subtitle":"Even in linear-quadratic control, any nonzero gradient ambiguity makes the optimal policy nonlinear.","key_machinery":"The load-bearing object is the inner adversary's maximization in the reduced GU-HJBI equation, namely $G(x,u,p)=\\sup_{\\lVert\\delta\\rVert\\le\\epsilon}\\big[(p+\\delta)^T f(x,u)+\\frac{\\eta}{2}\\lVert\\sigma(x,u)^T(p+\\delta)\\rVert^2\\big]$. This combines the drift sensitivity vector $v=f+\\eta\\sigma\\sigma^T p$ with a norm penalty, and its expansion $G=p^T f+\\frac{\\eta}{2}\\lVert\\sigma^T p\\rVert^2+\\epsilon\\lVert v\\rVert+O(\\epsilon^2)$ is what breaks the quadratic ansatz: the norm term is not a polynomial in $x$. The comparison principle is carried by the doubling-of-variables argument, with uniform ellipticity controlling the Hessian terms, and the perturbation analysis is carried by the linearized Lyapunov operator $\\mathcal{L}_{\\text{lin}}[W]=(\\nabla W)^T A_{\\text{eff},0}x+\\frac{1}{2}\\mathrm{Tr}(\\Sigma\\Sigma^T D^2 W)$.","core_discovery":"The central claim is that ambiguity about the value function gradient is a distinct, tractable form of uncertainty that changes the structure of robust control. The paper formulates a zero-sum game in which Nature chooses both a drift perturbation $h$ and a gradient perturbation $\\delta$, and argues that its dynamic programming form is the GU-HJBI equation (10). The paper proves a comparison principle and existence for this equation's viscosity solutions under a uniform ellipticity condition (Theorems 4.3 and 4.5). In the linear-quadratic setting it proves Proposition 5.2: for any $\\epsilon>0$ with non-degenerate data, the value function cannot be quadratic, because the perturbation Hamiltonian contributes a non-polynomial source term of the form $\\epsilon \\lVert A_{\\text{eff},0}x\\rVert$. The first-order correction $V_1$ then solves a linear PDE whose Feynman-Kac solution is generically non-polynomial, and the control correction $u_1 = -R^{-1}B^{T}\\nabla V_1$ is consequently nonlinear.","pith_inferences":["The paper leaves the game-theoretic underpinning of (10) implicit; supplying a rigorous dynamic programming principle would let the GU-HJBI equation serve as the definition of gradient-uncertainty robustness in its own right.","The non-polynomial norm source suggests value functions under gradient uncertainty can have kinks; numerical solvers and neural approximators that include nonsmooth features may track the solution better than smooth-only bases.","GURAC's penalty is a principled relative of gradient clipping and trust-region updates; a natural test is whether the regularization weight $\\lambda_R$ should scale with an estimate of the critic's gradient error rather than being a fixed hyperparameter.","The same construction applies to financial hedging where Greeks are approximate: adversarial perturbations of the Greeks would lead to nonlinear hedging strategies of the same kind."],"forward_implications":["Even in the simplest linear-quadratic control problem, any nonzero gradient uncertainty $\\epsilon$ makes the value function non-quadratic, so Riccati-equation methods cannot directly solve it; approximations must handle the non-polynomial correction.","For small $\\epsilon$, the value function correction solves a linear PDE with a norm source term, giving a concrete, numerically solvable path to compute robust policies beyond the standard LQ regulator.","In reinforcement learning, penalizing the actor with the norm of the drift sensitivity $f+\\eta\\sigma\\sigma^T\\nabla_x Q$ stabilizes training and reduces variance, as demonstrated in the Pendulum-v1 experiments.","Changing the geometry of the gradient-uncertainty set to $\\ell^2$, $\\ell^\\infty$, or a Mahalanobis ellipsoid changes the penalty to the corresponding dual norm, giving modelers a way to encode prior knowledge about gradient errors.","The framework extends the robust-control interpretation beyond relative entropy: the gradient perturbation attacks the agent's internal valuation, not the external model."],"supporting_citations":[{"why":"Supplies the viscosity-solution framework and Perron's method used to prove existence and the doubling-of-variables comparison principle.","marker":"Fleming and Soner [2006]"},{"why":"Provides the standard viscosity-solution machinery and technical conditions the comparison proof adapts.","marker":"Crandall et al. [1992]"},{"why":"Sets up the robust-control game with entropy penalties that this paper extends by adding gradient perturbations.","marker":"Hansen and Sargent [2008]"},{"why":"Gives the zero-sum differential game and Riccati-equation background for the LQ analysis.","marker":"Başar and Olsder [1998]"},{"why":"The TD3 algorithm that GURAC modifies; its clipped double-Q and delayed updates form the baseline and the empirical comparison.","marker":"Fujimoto et al. [2018]"}],"fun_headline_variants":["Gradient uncertainty breaks quadratic value functions","Nature's gradient jitter makes LQ control nonlinear","Nonzero gradient ambiguity makes optimal policy nonlinear","When value gradients are fuzzy, quadratic value fails","Gradient ambiguity upends classical LQ robust control"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that equation (10) is the dynamic programming equation of a well-defined zero-sum game with pointwise gradient perturbations, yet the paper never formally defines the game, its strategies, or a dynamic programming principle, so without that link the equation lacks the game-theoretic interpretation that motivates it.","fun_headline_variants_meta":{"raw":{"variants":["Gradient uncertainty breaks quadratic value functions","Nature's gradient jitter makes LQ control nonlinear","Nonzero gradient ambiguity makes optimal policy nonlinear","When value gradients are fuzzy, quadratic value fails","Gradient ambiguity upends classical LQ robust control"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000489,"raw_usage":{"total_tokens":2422,"prompt_tokens":972,"completion_tokens":1450,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":588,"completion_tokens_details":{"reasoning_tokens":1380}},"tokens_in":588,"tokens_out":1450,"duration_ms":12262,"temperature":1.0,"reasoning_tokens":1380,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:42:09.585724+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute, on a fine grid, the viscosity solution of (11) for the scalar LQ problem with $\\epsilon>0$ and compare it with the dynamic-programming value of an explicitly discretized version of the proposed zero-sum game; any discrepancy would show that equation (10) is not the game's dynamic programming equation. As a lighter check, test Proposition 5.2 by evaluating the second derivative of the numerically computed value function: a constant Hessian away from the origin would refute the claimed non-quadratic structure.","supporting_citations":[],"review_version":1}