{"id":"56995dde-27cb-48ea-85b6-a6981f85df66","arxiv_id":"2505.21842","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Solving a finite-horizon HJB equation with PINNs gives a practical, stabilizing-initializer-free route to infinite-horizon optimal control, with algorithms for horizon verification and extension.","lead":"This paper trains physics-informed neural networks on a finite-time version of the Hamilton-Jacobi-Bellman equation to approximate the optimal value function and controller for infinite-horizon nonlinear optimal control. The authors prove that the finite-horizon solution approaches the infinite-horizon one as the time horizon grows, and they supply algorithms to check and extend the horizon.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Lemma 2 is the load-bearing weak point: a vanishing tail integral over [T/2,T] does not by itself imply endpoint convergence x_T(T)→0, and both value and control convergence in Theorem 1 rest on this step.","rationale":"The reader's weakest-assumption analysis correctly identifies Lemma 2 as the load-bearing gap. My stress-test pass confirms that the inference from tail-integral decay to endpoint convergence is not justified by the assumptions given in the paper, and that Theorem 1's value convergence, uniform convergence, and the control convergence claim all depend on this step. The control-convergence proof is additionally incomplete, but it is secondary to the Lemma 2 gap because even the value-convergence part of the central claim is not secured by the current argument. The concern is substantive but fixable: adding coercivity of q, uniform bounds on the finite-horizon optimal trajectories, or a direct appeal to a known stability/detectability argument could repair the proof. The proposed concrete test isolates the logical step in a minimal setting where the paper's assumptions hold and would determine whether the missing implication is merely a proof gap or an actual failure of the theorem. I therefore agree with the reader's CONDITIONAL verdict rather than recommending rejection: the framework is plausible and the experimental evidence is supportive, but the main theorem requires additional assumptions or a substantially strengthened proof.","tokens_in":28458,"tokens_out":12820,"duration_ms":146486,"concrete_test":"Test Lemma 2 on the minimal scalar system ẋ = u with q(x) = x^2 e^{-x^2} (positive definite but non-coercive), R = 1, and zero terminal cost. Solve the finite-horizon optimal control problem via Pontryagin's principle (or a high-accuracy numerical two-point boundary value solver) for a sequence T → ∞, and check whether x_T(T) → 0 and whether V_T(x0,0) ≤ V*(x0) for all T. If x_T(T) fails to converge to 0 while V_T remains bounded by V*, Lemma 2 is false as stated; if it does converge, the proof must still be repaired by identifying the missing coercivity/regularity assumption that justifies the endpoint inference.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central convergence result, Theorem 1, depends on Lemma 2, which asserts lim_{T→∞} x_T(T) = 0 from the vanishing tail integral ∫_{T/2}^{T}(q(x_T)+u_T^T R u_T)dτ → 0. The proof says the energy of the state and control converge to zero in the steady state, 'which may hold only if lim x_T(T)=0', but this implication is not established under the stated assumptions. Vanishing of an L1 tail over a moving interval does not force pointwise convergence of the right endpoint: a trajectory could have a boundary layer near t=T whose width shrinks fast enough, or the state could move to a region where a non-coercive q is very small, unless one adds uniform bounds on x_T and u_T or coercivity of q. The paper only assumes q is positive definite and f,g locally Lipschitz (Section 2), which is insufficient for the endpoint inference. Lemma 3 then uses Lemma 2 to conclude V*(x_T(T))→0, and Lemma 4 and Theorem 1 inherit the gap. The control-convergence statement in Theorem 1 is also underproved: uniform convergence of V_T(·,0) to V* does not imply uniform convergence of ∇V_T, hence not of u_T(·,0)=-(1/2)R^{-1}g^T∇V_T. The same Lemma 2 is invoked again in the error-accumulation discussion after Theorem 3, so the gap propagates.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a PINN-based method for the infinite-horizon optimal control problem by training on the finite-horizon HJB equation (5) rather than the steady-state HJB equation (4), on the grounds that (5) has a unique solution. Section 4 claims that, as the horizon T tends to infinity, the finite-horizon value function V_T(·,0) and the associated controller u_T(·,0) converge uniformly to the infinite-horizon value function V* and optimal controller u* on a compact set Omega, for an arbitrary positive semidefinite terminal cost. Section 5 proposes a residual-based test for whether T is large enough, a horizon-extension procedure based on the dynamic programming principle, and an error-propagation analysis for approximate value functions. The paper closes with three numerical examples: a torsional pendulum, a system with known quartic value function, and a third-order system.","tokens_in":1882,"tokens_out":1993,"duration_ms":104192,"significance":"If the convergence theorem were rigorously established, the paper would offer a practical alternative to policy iteration and to direct PINN solution of the steady-state HJB: it would exploit the uniqueness of the finite-horizon HJB, would not require a stabilizing initial policy, and would work with non-polynomial basis functions. The horizon-extension idea in Theorem 2 and the error bound in Theorem 3 are useful contributions that appear largely correct under suitable regularity assumptions. However, the central convergence result currently rests on a gap in Lemma 2 and an underproved controller-convergence step in Theorem 1; these issues must be repaired before the main claim is acceptable.","major_comments":[{"comment":"The proof of Lemma 2 shows only that the tail integral from T/2 to T of q(x_T(tau)) + u_T^T R u_T vanishes as T tends to infinity, and then concludes that the endpoint x_T(T) converges to 0. This pointwise endpoint inference does not follow from a vanishing L1 tail over a moving interval: a boundary layer near t=T, or a trajectory spending most of the interval near the origin and then moving away, is not excluded by the stated assumptions. Positive definiteness of q does not provide coercivity, and no uniform bounds on x_T or u_T are assumed. Since Lemma 3 uses V*(x_T(T)) to 0 and Eq. (16), the pointwise convergence, the Dini-based uniform convergence in Lemma 4, and hence Theorem 1 all inherit this gap. The authors should either prove endpoint convergence under additional explicit assumptions or replace the argument.","section":"Section 4.1, Lemma 2"},{"comment":"The uniform convergence of u_T(·,0) to u* is asserted by saying that 'following the same arguments as in [4]' the trajectories converge uniformly and hence the controllers converge uniformly. Uniform convergence of the value functions does not imply uniform convergence of their gradients, and the controller is defined as u_T(·,0) = -(1/2)R^{-1}g^T grad V_T(·,0). A separate argument is needed to show that grad V_T(·,0) converges uniformly to grad V*, or otherwise to prove the uniform convergence of u_T directly. As written, the control-convergence half of Theorem 1 is not proved.","section":"Section 4.3, Theorem 1"},{"comment":"The proof of Lemma 5 uses the step that the maximum over x0 in Omega of gamma(x*_T(T)) tends to 0, i.e., uniform-in-initial-condition convergence of the optimal trajectories to the origin. Definition 1 only requires that the policy asymptotically stabilizes the system for each x0 in Omega, which is pointwise convergence and does not imply uniformity over the compact set Omega without an additional assumption such as uniform asymptotic stability. Since Lemma 5 is needed for the arbitrary-terminal-cost part of Theorem 1, this uniformity assumption should be stated explicitly or proved.","section":"Section 4.2, Lemma 5 and Eq. (20)"},{"comment":"The paragraph following Theorem 3 states that 'by Lemma 2, one has the limit as T' tends to infinity of x_{T'}(T'-T) equal to 0'. This is not a consequence of Lemma 2, which concerns the endpoint x_T(T) of the zero-terminal-cost problem with the same horizon. For a fixed T, x_{T'}(T'-T) is not an endpoint of a zero-terminal-cost optimal control problem, since the optimal policy over [0,T'] is not generally the optimal policy for the restricted interval [0,T'-T]. The claimed non-accumulation of approximation errors therefore requires a separate proof or an additional assumption.","section":"Section 5.3, after Theorem 3"},{"comment":"The residual E in Eq. (22) is proposed as a way to verify that V_T(·,0) is close to V*. However, the steady-state HJB equation (4) is known to have multiple solutions, as the paper itself emphasizes in Sections 1 and 3. A small value of E only indicates that V_T(·,0) nearly solves some solution of (4), not necessarily the optimal value V*. Thus the verification criterion needs an additional uniqueness or certificate argument, or it should be explicitly presented as a heuristic rather than a guaranteed test.","section":"Section 5.1, Eqs. (21)-(22)"}],"minor_comments":[{"comment":"The terminal condition is written as V_T(x,T)=psi(x), but the paper consistently uses V_T(·,0); it would help to state explicitly that V_T(x,t) is the value function with t time units remaining, so that the terminal condition and the evaluation at time 0 are unambiguous.","section":"Section 3, Eq. (5)"},{"comment":"Definition 1 defines admissibility for time-invariant feedback policies, whereas the finite-horizon problem (6) is minimized over time-varying policies u_T(x,t). The admissible class for the finite-horizon problem should be defined separately or the notation should be adjusted.","section":"Definition 1 and Eq. (6)"},{"comment":"The bound in Eq. (27) involves the approximation error e_T evaluated at states x_{T'}(T'-T) and hat x_{T'}(T'-T), which may lie outside the set Omega on which the PINN approximation error is controlled. Unless the trajectories are shown to remain in Omega, the bound is not directly usable as a quantitative error estimate.","section":"Section 5.3, Theorem 3"},{"comment":"The numerical comparison in Table 1 reports the mean squared residual (22) for different horizon lengths and for the direct steady-state HJB approach; it would be helpful to state the number of independent training runs and the variance of the reported MSE values, and to add axis labels and color bars to the figure panels.","section":"Section 6, Table 1 and Figures 1-7"}],"recommendation":"major_revision","confidential_remarks":"The main theorem is currently not proved because of the endpoint-convergence gap in Lemma 2 and the missing gradient-convergence step in Theorem 1. These seem repairable with additional standard assumptions, such as coercivity of q, uniform asymptotic stability, and a direct argument for controller convergence, rather than being fundamentally wrong. The horizon-extension results are a useful part of the paper and should be kept. I recommend major revision rather than rejection, provided the convergence claims are either rigorously proved under explicit assumptions or appropriately weakened."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I read the paper. Bottom line: the main idea (train a PINN on a finite-horizon HJB and extend the horizon) is worth taking seriously, but the proof of Theorem 1 has a real gap at Lemma 2 that needs work.\n\nActually new here is the horizon-extension scheme (Theorem 2), the verification residual (Section 5.1), and the error-propagation bound (Theorem 3). The uniform-approximation result itself is a known dynamic-programming fact, though not attributed; the packaging with PINNs and the extension machinery is a genuine step beyond their CDC 2023 paper.\n\nWhat the paper does well: it avoids the multi-solution steady-state HJB by solving a unique finite-horizon HJB, which is a natural fix. The experiments on the pendulum and the quartic-value system show the method works in practice, and the steady-state residual drops sharply after extension. Theorem 2 and Theorem 3 are correct and simple; the error bound in Theorem 3 is a clean, useful result.\n\nSoft spots: the stress-test note is right. Lemma 2 claims lim x_T(T)=0 from vanishing tail integral on [T/2,T]. That is not a valid inference without extra uniformity. The proof literally says \"which may hold only if...\" — that is a gap. Since Lemma 3 and Theorem 1 rest on it, the central convergence statement is under-proved. The control-convergence part of Theorem 1 is also asserted with a hand-wave to [4]; uniform convergence of V_T does not imply uniform convergence of grad V_T. These are fixable with standard regularity assumptions (coercivity of q, uniform bounds on trajectories, or a classical viscosity-solution convergence theorem), but as written they are real gaps.\n\nThe verification residual in Section 5.1 is also a heuristic: a small steady-state HJB residual means V_T(·,0) approximately satisfies equation (4), but that equation has many solutions; the residual cannot certify you are near V*. It is only a necessary condition, not sufficient. The authors should state this clearly. No code or data is provided, which makes reproduction harder.\n\nWho this is for: people working on PINN-based optimal control and on learning-based controllers that avoid policy iteration. It is a useful paper for that community despite the proof gap.\n\nRecommendation: I would send it to peer review. A good referee can identify the Lemma 2 issue and ask for a repaired proof. The numerical part is decent and the method is genuinely practical. It deserves a serious look, not a desk reject.","headline":"Finite-horizon PINN solver is a good idea, but the main convergence proof has a gap at Lemma 2 that must be fixed before publication.","tokens_in":29291,"tokens_out":2819,"would_cite":true,"duration_ms":27447,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["49L20","49K15","68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"Solving the finite-horizon HJB equation with PINNs provably recovers the infinite-horizon optimal control.","keywords":["physics-informed neural networks","optimal control","Hamilton-Jacobi-Bellman equation","finite-horizon approximation","nonlinear systems","value function","horizon extension","approximation error"],"falsifier":"Solve the finite-horizon HJB for a system with a known optimal value function (for instance the quartic-value system of Section 6.2) for a sequence of increasing horizons $T$, and record both the tail cost $\\int_{T/2}^{T}(q(x_T(\\tau)) + u_T(\\tau)^\\top R u_T(\\tau))\\,d\\tau$ and the endpoint state $x_T(T)$. If the tail cost tends to zero while $x_T(T)$ does not approach the origin, Lemma 2 is false and the uniform-convergence theorem needs extra assumptions. A concrete search target is a positive-definite running cost with dynamics whose closed-loop trajectories hover near a nonzero state for a long stretch and then decay sharply near the horizon, so that the windowed integral vanishes without the endpoint converging.","tokens_in":28217,"feed_emoji":"⚙️","tokens_out":11373,"duration_ms":104859,"temperature":0.7,"pith_summary":"This paper proposes a way to make physics-informed neural networks (PINNs) solve the infinite-horizon optimal control problem for nonlinear systems without running into the multiple-solution issue of the steady-state Hamilton-Jacobi-Bellman (HJB) equation. The idea is to train the PINN on the time-dependent, finite-horizon version of the HJB equation, which has a unique solution, instead of on the steady-state equation. The paper's central claim is that as the horizon grows, the finite-horizon value function at time zero converges uniformly to the true optimal value function, and the associated feedback control converges uniformly to the optimal policy, for any positive semidefinite terminal cost. It also proves a horizon-extension mechanism that reuses an already-computed value function and argues that approximation errors need not accumulate under repeated extensions, together with a residual-based test for deciding whether a chosen horizon is long enough. The paper presents the framework as a principled, guess-free route to learned optimal stabilization that requires neither an initial stabilizing controller nor iterative policy evaluation.","feed_headline":"Finite-horizon HJB unlocks PINNs for optimal control","feed_subtitle":"Replacing the multi-solution steady-state equation with a unique time-dependent one gives provable convergence.","key_machinery":"The central object is the finite-horizon value function $V_T(x,t)$, defined as the unique solution of the terminal-value HJB equation (5) and equivalently as the optimal value of the finite-horizon cost (6) with terminal penalty $\\phi$. It carries the argument because the steady-state HJB (4) is its formal stationary limit and because, unlike (4), equation (5) has a unique continuously differentiable solution, so a PINN trained on its residual (8) has a single target. The proof mechanism is the sandwich inequality (9): for any positive semidefinite $\\phi$, the value $V_T(\\cdot,0;\\phi)$ lies between the value with zero terminal cost and the value with terminal cost $\\phi+V^*$, both of which are shown to converge uniformly to $V^*$. The horizon-extension machinery in Theorem 2 is a second key element: it shows that the value for horizon $T'$ can be obtained from the value for horizon $T$ by solving a shorter companion PDE (24) over $[0,T'-T]$, which is a direct application of the principle of optimality.","core_discovery":"On the paper's own terms, the discovery is Theorem 1: for an arbitrary positive semidefinite terminal cost, the finite-horizon value function $V_T(\\cdot,0)$ of the time-dependent HJB equation (5) converges uniformly to the infinite-horizon optimal value function $V^*$ on any compact set $\\Omega$ as $T\\to\\infty$, and the corresponding control $u_T(\\cdot,0)$ converges uniformly to $u^*$. The argument funnels through a comparison inequality (9) that sandwiches the finite-horizon value for any terminal cost between the value with zero terminal cost and the value with terminal cost $\\phi+V^*$. The zero-terminal-cost side uses monotonicity in $T$ plus Dini's theorem to get uniform convergence; the $\\phi+V^*$ side uses the stabilizing property of the optimal policy to control the tail integral. This justifies replacing the multi-solution steady-state HJB (4) with the unique-solution finite-horizon HJB (5) inside a PINN training loop, and the paper adds a residual (22) to certify the horizon and a dynamic-programming extension (Theorem 2) to grow it without restarting.","pith_inferences":["The unproved endpoint-convergence step in Lemma 2 suggests that the uniform-convergence theorem as written requires an extra regularity assumption, such as uniform exponential decay of the finite-horizon optimal trajectories; the simulations may satisfy this informally even if the proof does not.","The residual-based horizon check (22) could be used online during training as a stopping or architecture-selection criterion, letting the network itself signal when the horizon or the network capacity must be increased.","If the endpoint-convergence gap is filled, the same template—embedding a multi-solution steady-state PDE in a unique-solution time-dependent companion equation—applies to other inverse and optimal-control problems beyond the specific HJB setting.","A natural next test is to extend the argument to zero-sum differential games, since the Isaacs equation suffers the same non-uniqueness pathology; the authors list exactly this as future work."],"forward_implications":["A PINN can be trained on the finite-horizon HJB (5) with any positive semidefinite terminal cost, and for a sufficiently large horizon $T$ the resulting value and control estimates are uniformly close to the true infinite-horizon optimal value and control on the compact set $\\Omega$.","The residual $E$ defined in (22) gives a computable test of whether a chosen horizon is large enough: a large residual signals either that $T$ should be increased or that the network architecture cannot represent the finite-horizon value accurately.","If the horizon is too small, the already-computed value function $V_T(\\cdot,0)$ can be used as a terminal cost in the shorter companion problem (23), whose solution coincides with the solution of the original longer-horizon problem (Theorem 2), so the horizon can be extended without solving from scratch.","Approximation errors from the initial solve do not necessarily accumulate when the horizon is extended multiple times: with a bounded error that vanishes at the origin, the extended-horizon error stays within the same bound (Theorem 3 and the discussion following it).","The approach avoids the three standard restrictions of policy iteration: it needs no prior knowledge of a stabilizing controller, does not require polynomial basis functions, and performs no iterative policy evaluations."],"supporting_citations":[{"why":"It defines the infinite-horizon optimal control problem, the steady-state HJB equation, and the optimal-control formula (3) that the whole method is built on.","marker":"[1]"},{"why":"The theorem's controller-convergence step explicitly follows the arguments in this reference, which supplies the technique for passing from uniformly convergent value functions to uniformly convergent controllers.","marker":"[4]"},{"why":"It supplies the PINN methodology and the observation that PINNs perform best when the underlying PDE has a unique solution, which motivates the finite-horizon formulation.","marker":"[15]"},{"why":"It is the prior direct PINN application to the steady-state HJB equation, providing the simulation baseline that shows the failure caused by non-unique solutions.","marker":"[16]"},{"why":"This is the authors' preliminary conference version, which served as the starting point; the present paper adds proofs, arbitrary terminal costs, and the robustness analysis.","marker":"[17]"},{"why":"It documents that the steady-state HJB equation has multiple solutions even in the linear-quadratic case, grounding the paper's motivation for replacing (4) with the unique-solution finite-horizon equation (5).","marker":"[18]"},{"why":"Dini's theorem from this reference is invoked in Lemma 4 to turn pointwise monotone convergence on the compact set $\\Omega$ into uniform convergence of $V_T(\\cdot,0)$ to $V^*$.","marker":"[20]"},{"why":"It provides the quartic-value-function benchmark system in Section 6.2, used to measure the approximation error against a known optimal solution.","marker":"[22]"}],"fun_headline_variants":["PINNs solve infinite-horizon control with finite-horizon HJB","Finite-horizon HJB trick lets PINNs solve infinite-horizon control","Avoiding multi-solution HJB: PINNs use finite horizon for unique value","Finite-horizon HJB converges to infinite-horizon optimum for PINNs","Replacing steady-state HJB with finite-horizon makes PINN optimal control tractable"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The proof assumes that because the tail integral of the running cost over $[T/2,T]$ vanishes as $T\\to\\infty$, the endpoint state $x_T(T)$ of the finite-horizon optimal trajectory must converge to the origin; this inference from a vanishing moving-window integral to pointwise endpoint convergence is asserted in Lemma 2 but not established.","fun_headline_variants_meta":{"raw":{"variants":["PINNs solve infinite-horizon control with finite-horizon HJB","Finite-horizon HJB trick lets PINNs solve infinite-horizon control","Avoiding multi-solution HJB: PINNs use finite horizon for unique value","Finite-horizon HJB converges to infinite-horizon optimum for PINNs","Replacing steady-state HJB with finite-horizon makes PINN optimal control tractable"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000617,"raw_usage":{"total_tokens":2900,"prompt_tokens":1018,"completion_tokens":1882,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":634,"completion_tokens_details":{"reasoning_tokens":1773}},"tokens_in":634,"tokens_out":1882,"duration_ms":15089,"temperature":1.0,"reasoning_tokens":1773,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:21:33.823592+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Solve the finite-horizon HJB for a system with a known optimal value function (for instance the quartic-value system of Section 6.2) for a sequence of increasing horizons $T$, and record both the tail cost $\\int_{T/2}^{T}(q(x_T(\\tau)) + u_T(\\tau)^\\top R u_T(\\tau))\\,d\\tau$ and the endpoint state $x_T(T)$. If the tail cost tends to zero while $x_T(T)$ does not approach the origin, Lemma 2 is false and the uniform-convergence theorem needs extra assumptions. A concrete search target is a positive-definite running cost with dynamics whose closed-loop trajectories hover near a nonzero state for a long stretch and then decay sharply near the horizon, so that the windowed integral vanishes without the endpoint converging.","supporting_citations":[{"cited_title":"Nearly optimal control laws for nonlinear systems with saturating actuators using a neural network HJB approach","cited_arxiv_id":null,"evidence_quote":"The theorem's controller-convergence step explicitly follows the arguments in this reference, which supplies the technique for passing from uniformly convergent value functions to uniformly convergent controllers."},{"cited_title":"Physics-informed neural net works: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations","cited_arxiv_id":null,"evidence_quote":"It supplies the PINN methodology and the observation that PINNs perform best when the underlying PDE has a unique solution, which motivates the finite-horizon formulation."},{"cited_title":"Physics-I nformed Neural Networks for Closed-Loop Guidance and Control in Aerospace Systems","cited_arxiv_id":null,"evidence_quote":"It is the prior direct PINN application to the steady-state HJB equation, providing the simulation baseline that shows the failure caused by non-unique solutions."},{"cited_title":"Principles of Mathematical Analysis","cited_arxiv_id":null,"evidence_quote":"Dini's theorem from this reference is invoked in Lemma 4 to turn pointwise monotone convergence on the compact set $\\Omega$ into uniform convergence of $V_T(\\cdot,0)$ to $V^*$."},{"cited_title":"Neural network approach to continuous-time d irect adaptive optimal control for partially unknown nonlinear systems","cited_arxiv_id":null,"evidence_quote":"It provides the quartic-value-function benchmark system in Section 6.2, used to measure the approximation error against a known optimal solution."}],"review_version":1}