REVIEW 2 major objections 6 minor 30 references
Reachability-Preserving Bellman Operator for the Discounted Reach-Cost Value Function: Uniting Hamilton-Jacobi Reachability and Reinforcement Learning
T0 review · 2 major / 6 minor · reviewed 2026-07-10 · grok-4.5
Pith's one-line read A non-additive Bellman operator for discounted reach-cost has a unique fixed point that is exactly the Hamilton-Jacobi reachability value function.
desk verdict Clean, usable completion of the HJ–RL bridge: a contractive non-additive Bellman operator whose fixed point exactly matches Choi’s semantics-preserving discounted reach value. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The reachability-preserving Bellman operator T_{σ,λ} (and its infinite-horizon counterpart). It replaces the usual sum of rewards by a min of a stopping branch and a discounted continuation branch; the discount factor e^{-λσ} makes the whole map a contraction whose unique fixed point is the true reachability value function.
What would settle it
On a low-dimensional system whose true Hamilton-Jacobi value function can be computed accurately (for example the double-integrator or Dubins-car examples already used in the paper), run fitted value iteration with the proposed operator and check whether the learned zero level set fails to align with the reference zero level set, or whether the pointwise residual fails to go to zero as the discretization parameters vanish.
Extended reading notes
Core claim
The discounted reach-cost value function that already appears in the Hamilton-Jacobi literature is exactly the unique fixed point of a non-additive Bellman operator. Discounting alone is enough to make that operator contractive on a complete space of bounded functions, so existence, uniqueness and convergence of value iteration follow, and the Hamilton-Jacobi and Bellman characterizations are identical.
Load-bearing premise
The system dynamics must be globally Lipschitz continuous in the state, uniformly in the control, and the reach-cost function must be bounded; without those two conditions the contraction proof and the uniqueness of the fixed point no longer hold on the whole space.
Editorial extensions
If this is right
- Learned reachability value functions can be interpreted as rigorous safety certificates rather than heuristic scores.
- Value iteration and fitted-value methods become legitimate numerical solvers for continuous-time reachable sets once the non-additive operator is used.
- The same contraction argument applies to both finite-horizon and infinite-horizon stationary problems, giving a uniform theory.
- High-dimensional systems that are intractable for grid-based Hamilton-Jacobi solvers become candidates for data-driven reachable-set computation while preserving semantics.
Reading between the lines
- The same construction should extend almost immediately to reach-avoid problems by swapping the min for a suitable max-min structure, yielding a single operator that encodes both safety and liveness.
- Once the contraction property is established, residual-based post-hoc certification methods already developed for additive Bellman operators become available for these non-additive reachability operators.
- The explicit separation of Bellman step σ from integration step Δt suggests a practical schedule that trades contraction speed against numerical consistency, which could be optimized automatically during learning.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper derives a non-additive Bellman operator for the semantics-preserving discounted reach-cost value function of Choi et al., proves that discounting renders the operator a contraction on a complete space of bounded functions, and shows that its unique fixed point coincides with the unique viscosity solution of the associated Hamilton–Jacobi variational inequality (finite- and infinite-horizon). Discrete-time consistent approximations and a fitted-value-iteration scheme are developed, and RL is interpreted as sample-based approximation of the same fixed-point equation. Numerical experiments on a double-integrator reach problem and a Dubins-car avoid problem show close agreement with HJ reference solutions and alignment of zero level sets.
Significance. If the results hold, the paper closes a genuine gap between HJ reachability and RL: prior discounted formulations either sacrificed exact reachability semantics for contraction (e.g., MDR) or preserved semantics on the HJ side without a matching Bellman fixed-point characterization. The operator-theoretic bridge (Definitions 5 and 7; Theorems 3–8) is clean, uses standard Banach and viscosity tools under classical Lipschitz/boundedness assumptions, and is complementary to concurrent travel-cost formulations. The explicit flagging of the open FVI convergence question (Remark 8) is a strength. The contribution is primarily theoretical; empirical support is limited to low-dimensional systems but is consistent with the claims that are actually proven.
major comments (2)
- Lemma 9 (and the parallel Lemma 12): the written contraction proof treats the rollout horizon as σ throughout and pulls out the factor e^{-λσ}, but Definition 5 uses h(τ)=min{τ,σ}. When τ<σ the continuation term lands on the fixed boundary Ψ(0,·)=g(·), so the difference is actually zero; Remark 4 correctly notes that the boundary condition is essential for uniform contraction near τ=0. The claim is true, but the proof as written does not case-split on h(τ). Please rewrite Lemma 9 (and the discrete analogue Lemma 14) with an explicit case analysis so that the argument is self-contained and matches Definition 5.
- Abstract, Introduction, and Section VIII: the paper repeatedly frames the contribution as uniting HJ reachability with reinforcement learning and as enabling “scalable, data-driven computation … in high-dimensional systems.” What is actually constructed is fitted value iteration with exact rollouts of a known dynamics model (explicitly identified as ADP in Section VIII), and Remark 8 correctly leaves open whether the composite “regress-then-apply-operator” map converges to the discrete fixed point. The operator-theoretic core (Theorems 3–8) does not depend on this claim, but the abstract and title overstate what is proven. Soften the RL/scalability language to match the proven fixed-point equivalence and the acknowledged open FVI question, or supply partial error bounds / high-dimensional evidence.
minor comments (6)
- Table I is a useful running reference; consider adding a one-line pointer to it at the first appearance of W_Bell / ĉW / Ψ_θ so readers do not miss the tier structure.
- Section IX-B: for the double integrator the infinite-horizon reachable set fills the domain, so zero-level-set comparison is impossible; this is explained, but a short finite-horizon double-integrator panel (or a reach-avoid variant) would make the zero-level-set claim more uniform across both examples.
- Eq. (42) / Definition 5: the notation U[0,h(τ)] vs U[0,σ] is slightly inconsistent with the later discrete operators; unify the control-interval notation.
- Proposition 3 writes V(x) rather than V(t,x); fix the argument list for consistency with Definition 1.
- Algorithms 1–2 are clear; a brief note on how U_d is chosen (grid density, effect on the min) would help reproducibility of the numerical scheme in Section VII.
- Related-work discussion of Hsu et al. and the concurrent travel-cost paper [27] is fair; a single sentence clarifying that the present operator is non-additive min-structured (not a running-cost integral) would further reduce possible confusion with additive discounted formulations.
Circularity Check
No significant circularity: the Bellman fixed-point and HJ equivalence are derived from the DPP plus Banach, not forced by definition or self-citation.
full rationale
The derivation chain is self-contained. The discounted reach-cost value functions W and W^∞ are defined directly from the dynamics and g (Defs. 1–4). The DPP (Props. 1–2) is obtained by splitting the infimum over time and using the semigroup property of the flow; the non-additive Bellman operators T_{σ,λ} and T^∞_{σ,λ} are then written down from those DPPs (Defs. 5, 7). Contraction follows from the elementary 1-Lipschitz property of min together with the factor e^{-λσ}<1 (Lems. 9, 12); existence/uniqueness of the fixed points is Banach’s theorem on the complete spaces B and B^∞ (Thms. 3, 5). That the true value functions are fixed points is immediate from the DPP, so uniqueness yields W = W_Bell and W^∞ = W^∞_Bell (Thms. 4, 6). The HJ characterizations are classical viscosity results (Thms. 1–2), with the avoid-case version already in Choi et al.; equivalence of the two characterizations is therefore just the observation that both objects equal the same W (Thms. 7–8). Self-citations to the authors’ concurrent travel-cost paper and residual-certification preprint appear only as complementary remarks and are not used in any proof. Experiments are pure numerical validation against an independent semi-Lagrangian HJ solver; no parameters are fitted and then re-predicted. The open FVI convergence question (Rem. 8) is correctly left open and does not affect the operator-theoretic claims. Hence the central results do not reduce to their inputs by construction.
Assumptions & free parameters
free parameters (3)
- discount rate λ
- Bellman step σ
- SIREN frequencies ω0 and network widths
assumptions (6)
- standard math Banach fixed-point theorem on complete metric spaces
- standard math Comparison principle / uniqueness of bounded continuous viscosity solutions for HJ variational inequalities
- domain assumption Global Lipschitz continuity of f in x, uniformly in u (Assumption 2)
- domain assumption Boundedness of the reach cost g (Assumption 8)
- domain assumption Sign calibration of g: g<0 inside target, g≥0 outside (Assumption 5)
- domain assumption Time-invariance of f and g (Assumptions 1, 6)
invented entities (1)
-
Reachability-preserving Bellman operator T_{σ,λ} / T^∞_{σ,λ}
independent evidence
Cite this review
Pith. "Pith review of Reachability-Preserving Bellman Operator for the Discounted Reach-Cost Value Function: Uniting Hamilton-Jacobi Reachability and Reinforcement Learning." pith.science (2026). https://pith.science/paper/HPKZLDQI
@misc{pith2026260707893,
author = {Pith},
title = {Pith review of: Reachability-Preserving Bellman Operator for the Discounted Reach-Cost Value Function: Uniting Hamilton-Jacobi Reachability and Reinforcement Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/HPKZLDQI}},
note = {Machine review of arXiv:2607.07893}
}
read the original abstract
Hamilton-Jacobi (HJ) reachability provides rigorous safety and reachability guarantees for continuous-time dynamical systems, but its numerical solution suffers from the curse of dimensionality. Deep reinforcement learning (DRL), by contrast, offers scalable sample-based methods. However, RL is typically built around additive cumulative rewards; whereas, reachability objectives are inherently non-additive. This mismatch makes a direct bridge between HJ reachability and RL nontrivial. Recent discounted formulations have either introduced contraction by altering the original reachability semantics, or preserved exact semantics on the HJ side without a corresponding Bellman fixed-point characterization. In this paper, we close this gap by building on a semantics-preserving discounted reach-based value function and deriving a non-additive Bellman operator whose unique fixed point exactly matches the value function in the HJ formulation. We prove that discounting makes this operator contractive, yielding existence, uniqueness, and convergence of value iteration. Furthermore, we establish the equivalence between the HJ and Bellman characterizations, and show that RL can be interpreted as a sample-based approximation scheme for the same fixed-point equation. This yields a principled and semantically exact connection between HJ reachability and RL, enabling learning-based methods to approximate reachability value functions while preserving their safety-critical meaning. As a result, the proposed framework opens the door to scalable, data-driven computation of reachable sets and safety certificates in high-dimensional systems. Numerical experiments demonstrate close agreement with HJ solutions, confirm preservation of reachability semantics via alignment of zero level sets, and support the interpretation of reinforcement learning as a sample-based solver of the proposed Bellman operator.
Figures
Reference graph
Works this paper leans on
-
[1]
Akametalu, Shromona Ghosh, Jaime F
Anayo K. Akametalu, Shromona Ghosh, Jaime F. Fisac, Vicenc Rubies- Royo, and Claire J. Tomlin. A minimum discounted reward hamil- ton–jacobi formulation for computing reachable sets.IEEE Transactions on Automatic Control, 69(2):1097–1103, 2024
work page 2024
-
[2]
OpenAI: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation.The International Journal of Robotics Research, 39(1):3– 20, 2020
work page 2020
-
[3]
Hamilton- jacobi reachability: A brief overview and recent advances
Somil Bansal, Mo Chen, Sylvia Herbert, and Claire J Tomlin. Hamilton- jacobi reachability: A brief overview and recent advances. In2017 IEEE 56th annual conference on decision and control (CDC), pages 2242–
-
[4]
Deepreach: A deep learning approach to high-dimensional reachability
Somil Bansal and Claire J Tomlin. Deepreach: A deep learning approach to high-dimensional reachability. In2021 IEEE International Conference on Robotics and Automation (ICRA), pages 1817–1824. IEEE, 2021
work page 2021
-
[5]
Martino Bardi, Italo Capuzzo Dolcetta, et al.Optimal control and viscosity solutions of Hamilton-Jacobi-Bellman equations, volume 12. Springer, 1997
work page 1997
-
[6]
Dimitri P. Bertsekas and John N. Tsitsiklis.Neuro-Dynamic Program- ming. Athena Scientific, Belmont, MA, 1996
work page 1996
-
[7]
Robust control barrier–value functions for safety- critical control
Jason J Choi, Donggun Lee, Koushil Sreenath, Claire J Tomlin, and Sylvia L Herbert. Robust control barrier–value functions for safety- critical control. In2021 60th IEEE Conference on Decision and Control (CDC), pages 6814–6821. IEEE, 2021
work page 2021
-
[8]
Jérôme Darbon and Stanley Osher. Algorithms for overcoming the curse of dimensionality for certain hamilton–jacobi equations arising in control theory and elsewhere.Research in the Mathematical Sciences, 3(1):19, 2016
work page 2016
Show all 30 references
-
[9]
Magnetic control of tokamak plasmas through deep reinforcement learning.Nature, 602(7897):414– 419, 2022
Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de Las Casas, et al. Magnetic control of tokamak plasmas through deep reinforcement learning.Nature, 602(7897):414– 419, 2022
2022
-
[10]
Bridging hamilton-jacobi safety analysis and reinforcement learning
Jaime F Fisac, Neil F Lugovoy, Vicenç Rubies-Royo, Shromona Ghosh, and Claire J Tomlin. Bridging hamilton-jacobi safety analysis and reinforcement learning. In2019 International Conference on Robotics and Automation (ICRA), pages 8550–8556. IEEE, 2019
2019
-
[11]
Hamilton-jacobi reachability estimation in reinforcement learning
Milan Ganai. Hamilton-jacobi reachability estimation in reinforcement learning. Master’s thesis, University of California, San Diego, 2024
2024
-
[12]
Hamilton-jacobi reachability in reinforcement learning: A survey.IEEE Open Journal of Control Systems, 3:310–324, 2024
Milan Ganai, Sicun Gao, and Sylvia L Herbert. Hamilton-jacobi reachability in reinforcement learning: A survey.IEEE Open Journal of Control Systems, 3:310–324, 2024
2024
-
[13]
Springer, 2003
Andrzej Granas, James Dugundji, et al.Fixed point theory, volume 14. Springer, 2003
2003
-
[14]
Scalable learning of safety guarantees for autonomous systems using hamilton-jacobi reachability
Sylvia Herbert, Jason J Choi, Suvansh Sanjeev, Marsalis Gibson, Koushil Sreenath, and Claire J Tomlin. Scalable learning of safety guarantees for autonomous systems using hamilton-jacobi reachability. In2021 IEEE International Conference on Robotics and Automation (ICRA), page...
2021
-
[15]
Safety and liveness guarantees through reach-avoid reinforcement learning.arXiv preprint arXiv:2112.12288, 2021
Kai-Chieh Hsu, Vicenç Rubies-Royo, Claire J Tomlin, and Jaime F Fisac. Safety and liveness guarantees through reach-avoid reinforcement learning.arXiv preprint arXiv:2112.12288, 2021
2021 arXiv
-
[16]
A hopf-lax formula in hamilton– jacobi analysis of reach-avoid problems.IEEE Control Systems Letters, 5(3):1055–1060, 2020
Donggun Lee and Claire J Tomlin. A hopf-lax formula in hamilton– jacobi analysis of reach-avoid problems.IEEE Control Systems Letters, 5(3):1055–1060, 2020
2020
-
[17]
Con- verse barrier certificates for finite-time safety verification of continuous- time perturbed deterministic systems.Systems & Control Letters, 209:106357, 2026
Yonghan Li, Chenyu Wu, Taoran Wu, Shijie Wang, and Bai Xue. Con- verse barrier certificates for finite-time safety verification of continuous- time perturbed deterministic systems.Systems & Control Letters, 209:106357, 2026
2026
-
[18]
On reachability and minimum cost optimal control
John Lygeros. On reachability and minimum cost optimal control. Automatica, 40(6):917–927, 2004
2004
-
[19]
Kavukcuoglu, David Silver, Andrei A
V olodymyr Mnih, K. Kavukcuoglu, David Silver, Andrei A. Rusu, J. Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, A. Fidjeland, Georg Ostrovski, Stig Petersen, Charlie Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, D. Kumaran, Daan Wierstra, S. Legg, and D....
2015
-
[20]
Error bounds for approximate value iteration.Proceedings of the National Conference on Artificial Intelligence (AAAI), 2005
Rémi Munos. Error bounds for approximate value iteration.Proceedings of the National Conference on Artificial Intelligence (AAAI), 2005
2005
-
[21]
Finite-time bounds for fitted value iteration.Journal of Machine Learning Research, 9:815–857, 2008
Rémi Munos and Csaba Szepesvári. Finite-time bounds for fitted value iteration.Journal of Machine Learning Research, 9:815–857, 2008
2008
-
[22]
Dual-objective reinforcement learning with novel hamilton-jacobi-bellman formulations.arXiv preprint arXiv:2506.16016, 2025
William Sharpless, Dylan Hirsch, Sander Tonkens, Nikhil Shinde, and Sylvia Herbert. Dual-objective reinforcement learning with novel hamilton-jacobi-bellman formulations.arXiv preprint arXiv:2506.16016, 2025
2025
-
[23]
Bellman value decomposition for task logic in safe optimal control
William Sharpless, Oswin So, Dylan Hirsch, Sylvia Herbert, and Chuchu Fan. Bellman value decomposition for task logic in safe optimal control. arXiv preprint arXiv:2602.19532, 2026
2026 arXiv
-
[24]
Mastering the game of go with deep neural networks and tree search.nature, 529(7587):484–489, 2016
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search.nature, 529(7587):484–489, 2016
2016
-
[25]
A general reinforcement learning algorithm that masters chess, shogi, and go through self-play.Science, 362(6419):1140–1144, 2018
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play.Science, 362(641...
2018
-
[26]
Mastering the game of go without human knowledge.nature, 550(7676):354–359, 2017
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge.nature, 550(7676):354–359, 2017
2017
-
[27]
Formalizing the relationship between hamilton-jacobi reachability and reinforcement learning.arXiv preprint arXiv:2601.08050, 2026
Prashant Solanki, Isabelle El-Hajj, Jasper van Beers, Erik-Jan van Kampen, and Coen de Visser. Formalizing the relationship between hamilton-jacobi reachability and reinforcement learning.arXiv preprint arXiv:2601.08050, 2026
2026 arXiv
-
[28]
van Beers, Erik-Jan van Kampen, and Coen C
Prashant Solanki, Isabelle El-Hajj, Jasper J. van Beers, Erik-Jan van Kampen, and Coen C. de Visser. Certifying hamilton-jacobi reachability learned via reinforcement learning.arXiv preprint arXiv:2602.16475, 2026
2026
-
[29]
Grandmaster level in starcraft ii using multi-agent reinforcement learning.nature, 575(7782):350–354, 2019
Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Math- ieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning.nature, 575(7782):350–354, 2019
2019
-
[30]
Inner-approximating reachable sets for polynomial systems with time-varying uncertainties
Bai Xue, Martin Fränzle, and Naijun Zhan. Inner-approximating reachable sets for polynomial systems with time-varying uncertainties. IEEE Transactions on Automatic Control, 65(4):1468–1483, 2019
2019
Reviewed July 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.