{"id":"7e4d82f1-28e7-4716-a30d-57ffbb9165be","arxiv_id":"2508.01718","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"A physics-informed neural network policy iteration framework solves stochastic optimal control problems via iterative linear PDE residual minimization with error-propagation bounds.","lead":"This paper describes a way to combine neural networks with an old optimization loop, policy iteration, to solve stochastic control problems. If it works as claimed, it gives a practical method with some mathematical guarantees for problems like stabilizing a cartpole or pendulum under noise.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Gradient-error control is the load-bearing condition: L2 PDE residual bounds from linear policy evaluation do not by themselves imply the gradient-error Lipschitz bounds required for contractive policy improvement.","rationale":"The reader's weakest assumption was essentially the same as mine: the network must attain a small PDE residual with sufficient control of value-gradient errors. I agree, and I sharpen the point: even perfect residual minimization in L2 is insufficient unless the PDE and training objective provide H1 control. The corrupted full text prevents checking whether the paper's assumptions include uniform ellipticity, boundary regularity, or a gradient term in the loss. The abstract alone does not establish these. Because the central claim is a guarantee, and the guarantee's key hypothesis is unverified, I would leave the reader's UNVERDICTED verdict unchanged rather than moving to accept or reject. A conditional acceptance would require the missing theorem hypotheses and the numerical gradient-error check described above.","tokens_in":11819,"tokens_out":3408,"duration_ms":46003,"concrete_test":"Take the 10D LQR benchmark, where the exact value function and optimal policy are known. Re-run PINN-PI while recording, at each iteration, the L2 PDE residual r_k = ||L_k V_nn - c_k|| and the gradient error g_k = ||grad V_nn - grad V*||. Check whether g_k / r_k stays bounded as the residual tolerance is decreased and whether the value error decreases exponentially across iterations. If the ratio grows without bound or policy improvement stalls before reaching the classical-PI rate, the Lipschitz propagation bound is not realized and the convergence claim is conditional on an unverified regularity and optimization assumption.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central convergence claim requires that the neural network solve each linear policy-evaluation PDE accurately enough that the greedy policy update remains a contraction. The abstract's support for this is an L2 residual bound plus 'explicit Lipschitz-type bounds' on how value-gradient errors propagate. The gap is that a small L2 residual controls the value error in L2, not in H1 or L-infinity. The policy improvement step is driven by the gradient of the value function; to keep the update contractive one needs ||grad(V_nn - V*)|| controlled by the residual (or by an explicitly tracked training error) uniformly over iterations and over the domain. For a uniformly elliptic second-order HJB operator, elliptic regularity can give such control, but stochastic control problems often have degenerate diffusion, and the stated 'mild conditions' are not verified in the corrupted text. If the Lipschitz constant in the propagation bound depends on an unverified ratio ||grad e||_L2 / ||e||_L2, or on a grid/network capacity that is not part of the theory, the claimed 'global exponential convergence of classical policy iteration' is not actually inherited. The phrase 'under mild conditions' hides exactly this training-accuracy and regularity assumption. This is not an internal contradiction, but it is the point where the argument's force is least secure.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a physics-informed neural network policy iteration (PINN-PI) algorithm for stochastic optimal control problems governed by second-order Hamilton-Jacobi-Bellman equations. The core ideas are to train a neural network at each iteration to approximate the value function of a fixed policy by minimizing the residual of the linearized policy-evaluation PDE, and then to use the linear structure to derive L2 error-control bounds and Lipschitz-type bounds on how value-gradient errors propagate to policy updates. The authors claim that the method inherits the global exponential convergence guarantees of classical policy iteration 'under mild conditions' and demonstrate it on stochastic cartpole, pendulum, and LQR problems in up to 10D. The submitted manuscript, however, is largely unreadable due to a severe encoding corruption in the full text, so the derivations, algorithm details, and experimental results cannot be inspected from the provided text.","tokens_in":12100,"tokens_out":3766,"duration_ms":45486,"significance":"If the theoretical claims are correct, the paper would make a valuable contribution to the emerging literature on rigorous error control for PINN-based methods in stochastic optimal control: systematic per-iteration L2 error bounds and explicit gradient-error propagation estimates would be a genuinely useful addition over heuristic PINN training, and the connection to classical policy iteration is a natural and credible theoretical anchor. The paper does not appear to rely on fitted constants or circular assumptions in its abstract-level claims, and the use of an external standard result (classical policy iteration convergence) is appropriate. However, the significance cannot currently be assessed because the full text is corrupted beyond readability, and the abstract alone leaves a load-bearing gap between L2 residual control and the gradient-error control needed for contractive policy improvement.","major_comments":[{"comment":"The body of the manuscript is unreadable: nearly all equations, theorem statements, algorithm pseudocode, and experimental tables are corrupted into mojibake. This is not a minor presentation issue; it prevents verification of every central claim. The abstract's assertions of L2 error control, Lipschitz-type bounds, and inherited exponential convergence cannot be checked against the actual derivations. The authors must resubmit a readable manuscript before the technical content can be evaluated.","section":"Full Text (throughout)"},{"comment":"The claim that the method 'inherits the global exponential convergence guarantees of classical policy iteration' requires that the neural-network approximations control sufficient derivatives of the value function, but the abstract only states L2 residual control at each policy evaluation step. A small L2 residual of the linearized HJB equation controls the value error in L2, not in H1 or L∞, and the policy improvement step depends on gradients of the value function. The paper must state the exact regularity, nondegeneracy, or training-accuracy assumptions under which the gradient-error propagation bounds hold; otherwise the convergence claim is not substantiated.","section":"Abstract / 'under mild conditions'"},{"comment":"The 'systematic L2 error control at each policy evaluation step' implicitly assumes that the neural network trained at that step actually attains a sufficiently small PDE residual in the norm used by the error bounds. This training-accuracy assumption is not stated in the abstract and appears to be hidden in the phrase 'under mild conditions.' If the Lipschitz constants in the propagation bounds depend on network capacity, grid resolution, or an unverified optimization success threshold, then the result is not a parameter-free inheritance of classical policy iteration. The precise assumption must be made explicit.","section":"Abstract"}],"minor_comments":[{"comment":"The phrase 'explicit Lipschitz-type bounds' should specify the exact norm in which value-gradient errors are measured (e.g., H1, W1,∞, or a weighted Sobolev norm) and whether the Lipschitz constants are explicitly computable or only existentially quantified.","section":"Abstract"},{"comment":"The experimental claims are qualitative ('demonstrate the effectiveness') with no numerical values, baselines, or error bars reported in the readable portion. Please include a table with quantitative comparisons to classical policy iteration or other PINN baselines, along with variance estimates.","section":"Abstract"},{"comment":"The descriptor 'high-dimensional' for LQR problems in 10D is overstated by current standards; consider either increasing the dimension or carefully qualifying the claim relative to existing stochastic control benchmarks.","section":"Abstract"},{"comment":"The corrupted encoding affects the reference list as well, making it impossible to verify the novelty claims relative to existing PINN policy iteration literature. Please ensure that the source compiles to a clean PDF with an intact bibliography.","section":"Full Text (bibliography)"}],"recommendation":"major_revision","confidential_remarks":"The main obstacle is the corrupted submission: the paper cannot be reviewed in its current form. The technical concern raised by the stress-test—that L2 residual control does not by itself imply the gradient-error control needed for contractive policy improvement—is likely to be the central point in the next round. I recommend returning the paper to the authors to resubmit a readable PDF and to address the gradient-error assumptions explicitly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper's core promise—a neural policy-iteration solver for stochastic optimal control with per-iteration L2 error control and explicit Lipschitz bounds on gradient-error propagation—is a genuine extension of the deterministic PINN line, and the 10D LQR benchmark is a reasonable stress test. That much is worth taking seriously. I can't verify more: the full text on arXiv is mojibake, so the derivations and experiments are not inspectable in this version.\n\nWhat I can assess is the abstract's logic. The interesting new claim is the gradient-error propagation bound. That is also where the argument is least secure. A small L2 PDE residual controls the value error in L2, but the policy improvement step is driven by gradients. To keep the policy update contractive, you need gradient error control, either from elliptic regularity or from an explicit training-error term. For degenerate-diffusion stochastic control, regularity may not save you, and the phrase 'under mild conditions' is doing a lot of work. The stress-test note is right: the load-bearing condition is that the trained network achieves a residual small enough in the norm that matters, and the abstract does not show that. This is not an internal contradiction, just the soft spot where the convergence guarantee could silently depend on an unverified ratio like ||∇e||/||e||.\n\nCredit where due: the method is clearly stated, the benchmarks are non-trivial, and the theoretical framing (linear policy evaluation -> L2 control -> Lipschitz propagation) is the right way to make the argument. The absence of visible baselines and error bars in the abstract is a concern, but that's normal for an abstract; the full text may address it.\n\nBottom line: this belongs in peer review, with one specific instruction to the referee—check whether the training residual is controlled in a norm (H1 or stronger) sufficient for the contraction argument, and whether the 'mild conditions' cover degenerate diffusion. For a reader working on PINN/HJB solvers, this is worth reading once a clean version exists. I wouldn't cite it yet.\n\nRecommendation: send to review, not desk reject.","headline":"A plausible and genuinely new PINN policy-iteration method for stochastic HJB, whose convergence claim hinges on gradient-error control that the abstract does not establish.","tokens_in":12537,"tokens_out":2414,"would_cite":false,"duration_ms":25143,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Policy iteration with neural-network value solves can solve stochastic optimal control with provable error control and inherited exponential convergence.","keywords":["stochastic optimal control","Hamilton-Jacobi-Bellman equation","policy iteration","physics-informed neural networks","PDE residual error bounds","linear quadratic regulator","convergence guarantees","value function approximation"],"falsifier":"On a stochastic LQR problem with a known exact value function, compute the paper's claimed $L^2$ error bound and the true error after every iteration; an iteration where the bound fails, or where the residual shrinks while the policy's realized cost worsens, would show the error-control assumptions do not hold.","tokens_in":11667,"feed_emoji":"🧠","tokens_out":6785,"duration_ms":74191,"temperature":0.7,"pith_summary":"The paper is trying to establish that policy iteration works when each value-function evaluation is replaced by a neural-network solve, provided the solve targets a linear PDE with controlled residual. This matters because stochastic optimal control problems are usually governed by a nonlinear Hamilton-Jacobi-Bellman equation that is hard to solve, while fixing a policy makes the equation linear and therefore amenable to a posteriori error control. The authors derive explicit $L^2$ error bounds for each evaluation step and Lipschitz-type bounds that show how value-gradient errors affect the policy update, so the method inherits the global exponential convergence of classical policy iteration under mild conditions. They demonstrate the approach on stochastic cartpole and pendulum benchmarks and on linear quadratic regulator problems in up to 10 dimensions.","feed_headline":"Stochastic control solved by neural policy iteration with error bounds","feed_subtitle":"Each step solves a linear PDE, so value errors stay bounded and classical exponential convergence carries over.","key_machinery":"The central object is the fixed-policy value equation: for a policy $\\pi$, the value function $V^\\pi$ solves a linear second-order PDE $\\mathcal{L}_\\pi V^\\pi = \\ell_\\pi$, where $\\mathcal{L}_\\pi$ contains the drift and diffusion terms of the controlled stochastic process. The algorithm trains a neural network $\\hat{V}$ to minimize the PDE residual $\\|\\mathcal{L}_\\pi \\hat{V} - \\ell_\\pi\\|_{L^2}$; linearity is what makes the residual a valid certificate, yielding an $L^2$ bound on $V^\\pi - \\hat{V}$ and Lipschitz-type bounds on the map from value-gradient error to policy change. Those bounds are what connect a purely numerical solve to the classical policy iteration convergence theorem.","core_discovery":"The central claim is that the linear structure of the fixed-policy PDE is enough to make neural policy iteration reliable: each iteration trains a network to minimize the residual of a linear second-order equation satisfied by the value of the current policy, and the residual controls the $L^2$ error in the value approximation. From that control, the paper derives explicit Lipschitz-type bounds quantifying how errors in the value gradient propagate into the policy-improvement step. Under mild regularity and training-accuracy conditions, the loop therefore retains the global exponential convergence guarantee of classical policy iteration rather than relying on heuristics. The paper reports that this behavior is observed on stochastic cartpole, pendulum, and 10-dimensional LQR benchmarks.","pith_inferences":["The same residual-based certificate could be used as an adaptive training guide, for example by sampling more collocation points where the residual is large; the paper stops at using the residual for error control, so this is an extension.","Because the convergence argument depends on linearity rather than on neural networks, other linear approximators such as finite elements or sparse grids could plug into the same policy-evaluation step.","The practical bottleneck is likely high-dimensional residual minimization: in 10D the $L^2$ error control may be hard to verify thoroughly, so a natural next test is whether the bounds stay informative above 10 dimensions.","One could set a target value-gradient accuracy before an iteration and derive the required network training budget from the Lipschitz bounds, turning the analysis into an a priori stopping rule."],"forward_implications":["Each policy-evaluation step can be monitored: the PDE residual gives a computable bound on value-function error, so users can certify approximation quality during training.","Value-gradient errors have a quantified effect: the Lipschitz-type bounds say how much the improved policy can change when the learned gradient is off, making the method interpretable.","The method inherits global exponential convergence of classical policy iteration under mild conditions, so the iteration count to a target accuracy is controlled by problem constants rather than by network architecture.","Stochastic control benchmarks including cartpole, pendulum, and up to 10-dimensional LQR are handled by the same framework, indicating the approach is not limited to small or deterministic problems."],"supporting_citations":[],"fun_headline_variants":["Neural policy iteration with error bounds for stochastic control","Physics-informed neural policy iteration: error control for HJB problems","Stochastic control with neural policy iteration: linear PDE error guarantees","Provable error bounds make neural policy iteration viable for stochastic control","Neural policy iteration with explicit Lipschitz error bounds for stochastic control"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The guarantee assumes the neural network actually drives the equation residual and its gradient small enough at every step; the linear-PDE analysis does not itself ensure that training succeeds.","fun_headline_variants_meta":{"raw":{"variants":["Neural policy iteration with error bounds for stochastic control","Physics-informed neural policy iteration: error control for HJB problems","Stochastic control with neural policy iteration: linear PDE error guarantees","Provable error bounds make neural policy iteration viable for stochastic control","Neural policy iteration with explicit Lipschitz error bounds for stochastic control"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000462,"raw_usage":{"total_tokens":2259,"prompt_tokens":839,"completion_tokens":1420,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":455,"completion_tokens_details":{"reasoning_tokens":1334}},"tokens_in":455,"tokens_out":1420,"duration_ms":12171,"temperature":1.0,"reasoning_tokens":1334,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T05:24:45.431646+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a stochastic LQR problem with a known exact value function, compute the paper's claimed $L^2$ error bound and the true error after every iteration; an iteration where the bound fails, or where the residual shrinks while the policy's realized cost worsens, would show the error-control assumptions do not hold.","supporting_citations":[],"review_version":1}