Pith. sign in

REVIEW 3 major objections 4 minor 22 references

Approximating Safety Feedback Without a Safety Oracle via Model Predictive Control

T0 review · 3 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that safety can be certified by reversibility: an action is safe if the agent can plan a path back to its previous state, and it demonstrates a training algorithm, RL-SA VMPC, that uses MPPI to enforce this check.

desk verdict Novel combination of reversibility and MPC shielding, but the claimed safety guarantee doesn't survive the δ relaxation and missing hyperparameters. read the letter →

arxiv 2510.20955 v2 pith:UI36QQI7 submitted 2025-10-23 cs.LG cs.RO

Jeff Pflueger , Michael Everett This is my paper · ORCID
classification cs.LGcs.RO
keywords safetyinreinforcementlearningmodelpredictivepathintegralcontrolreversibilitypositiveinvarianceblack-boxdynamicsshieldingconstrainedMarkovdecisionprocesssafeexploration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that many safety constraints that are hard to specify can be captured by a single property: once the agent enters the unsafe region, it can never leave. If that is true, safety can be checked by reversibility: a candidate action is safe if the agent can plan a way back to the state it started from. The proposed algorithm, RL-SA VMPC, uses Model Predictive Path Integral control through a black-box simulator to look for such a return path before executing any action. If no return path is found, the trajectory is aborted. Experiments in two environments show the method aborts before all unsafe actions while matching the training reward of an unconstrained baseline.

What carries the argument

The load-bearing object is the MPPI controller used as a feasibility oracle. MPPI is a sampling-based model predictive control routine that searches over many action sequences through the black-box dynamics. It converts an intractable infinite-horizon reachability problem into a finite-horizon, tolerance-relaxed version, making reversibility checkable with only query access to the transition function. The invariance of the unsafe set is what lets a found return path imply that the new state is in the same safe set as the old state.

What would settle it

A counterexample environment where a constraint violation can be undone, say a robot that tips over but can right itself. If the MPPI planner returns a path from the tipped-over state back to the previous state, RL-SA VMPC will execute the violating action and report zero violations, showing that reversibility to a previous safe state does not certify safety when the invariance assumption fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that, under the assumption that the unsafe set is positive invariant, a state is safe exactly when there exists a reversibility to a state already known safe. The algorithm operationalizes this: at each step, sample a proposed action, project it through the simulator, and run MPPI to find a sequence of actions back to the previous state. Existence of the return plan certifies the state safe; nonexistence aborts the episode. The paper reports that in a continuous cartpole and a 2D navigation task, RL-SA VMPC produces zero constraint violations and reward comparable to baseline PPO, and in navigation it outperforms a resampling shield with perfect constraint knowle

Load-bearing premise

The unsafe set is permanent: once an agent enters X_unsafe, no sequence of actions can ever bring it back to a safe state.

Editorial extensions

If this is right

  • Safety verification becomes possible without hand-specified constraints or labeled safety data; a queryable simulator is sufficient.
  • In deterministic environments, training can be shielded from all constraint violations because a successful return plan is a certificate of safety.
  • The method can outperform one-step safety shields in environments with attraction basins, because the return plan looks ahead over the whole horizon.
  • The training reward of a baseline policy is approximately preserved, so the shield does not drastically restrict exploration.
  • The approach is limited to deterministic dynamics; stochastic settings are left to future work.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the invariance assumption fails for a real safety constraint, the method can produce false positives: a temporary violation that can be reversed would be certified safe.
  • The reversibility requirement may be overly conservative in systems with one-way safe regions, where a state is actually safe but no path back exists, leading to unnecessary aborts.
  • The safety tolerance delta introduces boundary risk near the safe/unsafe frontier, so small violations could be masked by a return path that only gets within delta of the previous state.
  • A natural extension is to test the method in asymmetric or non-holonomic environments, where the inability to reverse may make the filter abort frequently and reveal how much performance is lost to conservatism.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes RL-SA VMPC, a safe RL method that replaces an explicit safety oracle with a reversibility check implemented by Model Predictive Path Integral control (MPPI). The agent's action is first propagated through a black-box simulator, and MPPI then tries to find a plan from the resulting state back to the previous trajectory state. Under the assumption that the unsafe set is positive invariant, existence of such a return plan is claimed to certify that the current state is safe. Experiments on a continuous cartpole and a 2D navigation task compare RL-SA VMPC with baseline PPO and a perfect-knowledge resampling shield, reporting zero constraint violations and comparable reward.

Significance. The core idea—using reversibility as a proxy for safety with only black-box dynamics—is genuinely interesting and could reduce reliance on hand-specified constraints in simulation-based RL. The comparison against a perfect-knowledge shield is a useful sanity check, and the two environments are appropriate stress tests for the method. However, the paper's central safety guarantee is not supported by the implemented algorithm: the exact feasibility problem in Eq. (1) is relaxed to Eq. (2) with finite horizon T and tolerance δ, and MPPI is a stochastic sampler without completeness guarantees. These relaxations directly undermine the claimed guarantee, not just its scope. The paper would need either a formal analysis of the relaxed problem or a reframing as an empirical heuristic.

major comments (3)
  1. [§IV-B, Eq. (2) and the paragraph after it] The claim that 'this problem should never produce a false positive set of actions' is not valid for the implemented relaxation. Eq. (2) replaces the exact reachability condition ∃k≥0 : x_k = x with ∃k≤T : ‖x_k − x‖₂ ≤ δ, and it does not constrain the path to avoid X_unsafe. If the δ-ball around the previous safe state x intersects X_unsafe, a candidate state x′ that is actually unsafe can be classified safe as long as MPPI returns a trajectory ending near x. Moreover, MPPI is a sampling-based optimizer with no completeness: it may return a 'solution' when none exists in the exact sense, or fail to find an existing one. The sentence 'In an environment with deterministic dynamics, this problem should never produce a false positive' refers to the exact problem (1), not to (2) that is actually solved. This is load-bearing because the paper's central guarantee—'guaranteed to be outside the un
  2. [§IV-A (Definition 4.3) and §V-A (Continuous Cartpole)] The invariance assumption is stated clearly, but it is not satisfied by the experimental testbeds, and this mismatch undermines the empirical validation. In Continuous Cartpole, exceeding the 12° angle bound does not make the state uncontrollable or 'broken beyond repair'; a recovery action that brings the pole back within bounds likely exists. Under Definition 4.3, such a state would be classified safe because a reversibility to the previous safe state exists, even though it violates the ground-truth constraint. Thus the filter can, in principle, approve unsafe actions whenever the unsafe set is not truly invariant. The reported zero violations may be an artifact of MPPI's conservatism or the particular δ/T choices rather than evidence that the reversibility logic is sound. The authors should either use environments with genuinely invariant unsafe sets, or explicitly acknowledge and ana
  3. [§IV-B and Algorithm 2] The safety-critical hyperparameters—δ, T, N_samples, and the MPPI sampling covariance and cost weights—are not reported anywhere. The text asserts 'choosing a small enough δ will limit the impact' of boundary violations, but without the actual values and a sensitivity study, this claim is not reproducible. Since δ and T directly determine what counts as 'returning' to the previous state, the regime in which false positives are excluded is unspecified. This is particularly important because the experiments report zero violations; the reader cannot assess whether this is a robust property or a fine-tuned result. Please report these values and provide a sensitivity analysis, at least for δ and T.
minor comments (4)
  1. [Abstract vs. §IV-B] The abstract says the returned state is 'guaranteed to be outside the unsafe set,' but the paper later admits that 'constraint violations are theoretically possible' due to the δ relaxation. This inconsistency should be resolved by using qualified language throughout.
  2. [Algorithm 2, line 5 and Fig. 2] The variable U_safe is used both as the set of actions returned by MPPI and as a Boolean indicator (empty/nonempty). Please clarify the return type and define how U_safe is constructed from an MPPI plan. Figure 2 labels 'Usafe=∅' and 'Usafe≠∅' but the text never references the figure.
  3. [Sec. V-A] In the Two Dimensional Navigation environment, the state is written as 'x=(p_x, y_x, p_goalx, p_goaly)'; the second component should presumably be p_y rather than y_x. This typo makes the state definition confusing.
  4. [References] Reference [3] contains a typo: 'CONSERV ATIVE SAFETY CRITICS FOR EXPLORATION' should be 'Conservative Safety Critics for Exploration.'

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reversibility-based safety argument is self-contained, and the MPPI relaxations are disclosed limitations rather than hidden fits.

full rationale

The paper's derivation is self-contained: safety is defined as reversibility to a known safe state (Definition 4.3) under the stated invariance of X_unsafe, and the algorithm builds an inductive chain from an assumed-safe initial state. The MPPI-based check (Eq. 2) is an approximate feasibility test for the exact reversibility problem (Eq. 1); the paper explicitly concedes that the δ-tolerance means the backup may not return to the exact previous state and may create boundary violations. That is a correctness/completeness limitation, not a circular construction: δ and T are not fitted to the violation metric, and the experiments are scored against ground-truth constraints. No parameter is renamed as a prediction, no load-bearing self-citation exists (the reversibility references [11], [17] are external prior work), and no uniqueness or ansatz is imported from the authors' own prior results. The central claim therefore does not reduce to its own inputs by construction.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

No new physical entities are introduced. The method rests on several domain assumptions (invariant unsafe set, deterministic dynamics, accurate simulator, known safe start) and on several unreported hyperparameters that determine the safety verdict. The largest unstated burden is that MPPI is treated as a reliable feasibility oracle despite being a stochastic sampler.

free parameters (5)
  • δ (safety tolerance) = not reported
    In Eq. (2), a backup plan is accepted if ∥x_k − x∥ ≤ δ; larger δ can admit boundary violations while smaller δ increases aborts. The value directly controls what counts as 'safe'.
  • T (planning horizon) = not reported
    Eq. (2) limits the number of backward steps; too small a horizon causes false negatives (unnecessary aborts), too large increases computation.
  • N_samples (MPPI samples) = not reported
    Algorithm 2 samples up to N_samples actions from the policy; too few samples may fail to find a feasible recovery path.
  • λ (cartpole reward aggression) = not reported
    Eq. (3) defines R(x) = 1 + λθ, incentivizing the agent to approach the pole-angle limit; λ changes exploration risk and is not reported.
  • MPPI hyperparameters (sampling covariance, iterations, cost weights) = not reported
    MPPI is the feasibility solver, but the paper does not give its cost design, sampling variance, or iteration count, which are essential to reproduce the safety check.
assumptions (5)
  • domain assumption X_unsafe is invariant
    Section IV-A states this as a key assumption. Reversibility to a previous safe state is only a safety proof if the unsafe set cannot be exited.
  • domain assumption Deterministic transition dynamics f_dyn
    The problem statement defines f_dyn as deterministic, and Section IV-B says 'In an environment with deterministic dynamics, this problem should never produce a false positive.' Stochastic dynamics are left to future work.
  • domain assumption The black-box simulator accurately models the true dynamics and latent constraints
    The algorithm queries f_dyn as a stand-in for the real environment; if the simulator misses a constraint, the reversibility check is not meaningful.
  • domain assumption Initial state x0 is known safe
    Algorithm 1 requires X0 as the initial state space, and the reversibility chain works backward to x0. If x0 is unsafe, the whole chain loses its anchor.
  • ad hoc to paper MPPI reliably solves the feasibility problem in Eq. (2)
    MPPI is a stochastic optimization algorithm, not an exact feasibility checker. The paper does not prove that a non-empty U_safe implies a true trajectory within δ; finite samples and approximate convergence can produce false positives.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Approximating Safety Feedback Without a Safety Oracle via Model Predictive Control." pith.science (2026). https://pith.science/paper/UI36QQI7

@misc{pith2026251020955,
  author       = {Pith},
  title        = {Pith review of: Approximating Safety Feedback Without a Safety Oracle via Model Predictive Control},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UI36QQI7}},
  note         = {Machine review of arXiv:2510.20955}
}
read the original abstract

Safe decision-making algorithms for control of mobile robots often require the existence of feedback to verify the safety of proposed actions. This feedback is assumed to be directly available during the development or deployment of the control system. It can take the form of either an explicit constraint formulation or a set of hand-labeled safety data, both of which can be inaccurate or time consuming to produce. Many recently developed simulators can handle complex interactions and varied environments. These environments have implicit safety constraints that may be hard to model. By leveraging one of these simulators, we can construct a proxy for a safety function that bypasses the need for hand designed feedback in capturing these constraints. We present an algorithm that approximates safety by using reversibility and a positive-invariance assumption on the unsafe state space. This method employs the Model-Predictive Path Integral algorithm (MPPI) to establish this reversibility and verify a proposed action. First the action is projected via the simulator to a future state. Then if MPPI can find a path back to a previous state in the trajectory, that state is guaranteed to be outside the unsafe (positive invariant) set. Experimental results demonstrate that the proposed algorithm can approximate the performance of a safety oracle while avoiding classification of unsafe states as safe.

Figures

Figures reproduced from arXiv: 2510.20955 by the authors.

Figure 1
Figure 1. Illustration of RL-SAVMPC. The agent is attempting [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Block diagram of RL-SAVMPC. At state x, the algorithm samples an action from the RL Policy fθ. It then propagates the action forward to get the induced state x ′ . The safety planner tries to find a set of actions Usaf e which return the agent to the previous state. If such a sequence exists, the original action is sent to the agent. If they do not exist, RL-SAVMPC resamples from fθ [9] in a deterministic MDP on the… view at source ↗
Figure 3
Figure 3. Reward results from both the Continuous Cartpole and Two Dimensional Navigation environments. These results are [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Constraint violations and aborts for each training environment. These results are averaged over 10 seeds, and smoothed. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 6 linked inside Pith

  1. [1]

    Sutton and Andrew G

    Richard S. Sutton and Andrew G. Barto.Reinforcement learning: an introduction. Adaptive computation and machine learning. MIT Press, Cambridge, Mass, 1998

    1998

  2. [2]

    Routledge, 1 edition, March 1999

    Eitan Altman.Constrained Markov Decision Processes: Stochastic Modeling. Routledge, 1 edition, March 1999

    1999

  3. [3]

    CONSERV ATIVE SAFETY CRITICS FOR EXPLORATION

    Homanga Bharadhwaj, Aviral Kumar, Nicholas Rhinehart, Sergey Levine, Florian Shkurti, and Animesh Garg. CONSERV ATIVE SAFETY CRITICS FOR EXPLORATION. 2021

    2021

  4. [4]

    Learning to be Safe: Deep RL with a Safety Critic, October 2020

    Krishnan Srinivasan, Benjamin Eysenbach, Sehoon Ha, Jie Tan, and Chelsea Finn. Learning to be Safe: Deep RL with a Safety Critic, October 2020. arXiv:2010.14603 [cs]

    arXiv 2020

  5. [5]

    When to Ask for Help: Proactive Interventions in Autonomous Reinforcement Learning

    Annie Xie, Fahim Tajwar, Archit Sharma, and Chelsea Finn. When to Ask for Help: Proactive Interventions in Autonomous Reinforcement Learning. arXiv, October 2022. arXiv:2210.10765 [cs]

    arXiv 2022

  6. [6]

    Safe Exploration in Finite Markov Decision Processes with Gaussian Pro- cesses

    Matteo Turchetta, Felix Berkenkamp, and Andreas Krause. Safe Exploration in Finite Markov Decision Processes with Gaussian Pro- cesses. 2016

    2016

  7. [7]

    Safe Reinforcement Learning in Con- strained Markov Decision Processes, August 2020

    Akifumi Wachi and Yanan Sui. Safe Reinforcement Learning in Con- strained Markov Decision Processes, August 2020. arXiv:2008.06626 [cs]

    arXiv 2020

  8. [8]

    Safe Reinforcement Learning via Shielding.AAAI, 32(1), April 2018

    Mohammed Alshiekh, Roderick Bloem, R ¨udiger Ehlers, Bettina K¨onighofer, Scott Niekum, and Ufuk Topcu. Safe Reinforcement Learning via Shielding.AAAI, 32(1), April 2018

    2018

Show all 22 references
  1. [9]

    Safe Reinforcement Learning with Nonlinear Dy- namics via Model Predictive Shielding

    Osbert Bastani. Safe Reinforcement Learning with Nonlinear Dy- namics via Model Predictive Shielding. In2021 American Control Conference (ACC), pages 3488–3494, May 2021. ISSN: 2378-5861

    2021

  2. [10]

    Shuo Li and Osbert Bastani. Robust model predictive shielding for safe reinforcement learning with stochastic dynamics.2020 IEEE International Conference on Robotics and Automation (ICRA), pages 7166–7172, 2019

    2020

  3. [11]

    Don’t Do Things You Can’t Undo: Reversibility Models for Generating Safe Behaviours

    Maarja Kruusmaa, Yuri Gavshin, and Adam Eppendahl. Don’t Do Things You Can’t Undo: Reversibility Models for Generating Safe Behaviours. InProceedings 2007 IEEE International Conference on Robotics and Automation, pages 1134–1139, April 2007. ISSN: 1050- 4729

    2007

  4. [12]

    Cambridge University Press, 2017

    Francesco Borrelli, Alberto Bemporad, and Manfred Morari.Predic- tive control for linear and hybrid systems. Cambridge University Press, 2017

    2017

  5. [13]

    Provable Safe Reinforcement Learning with Binary Feedback

    Andrew Bennett, Dipendra Misra, and Nathan Kallus. Provable Safe Reinforcement Learning with Binary Feedback. InProceedings of the 26th International Conference on Artificial Intelligence and Statistics (AISTATS) 2023, volume 206, 2023

    2023

  6. [14]

    Long- Term Safe Reinforcement Learning with Binary Feedback

    Akifumi Wachi, Wataru Hashimoto, and Kazumune Hashimoto. Long- Term Safe Reinforcement Learning with Binary Feedback. InProceed- ings of the AAAI Conference on Artificial Intelligence, volume 38, pages 21656–21663, March 2024

    2024

  7. [15]

    Conservative q-learning for offline reinforcement learning

    Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. NIPS ’20, Red Hook, NY , USA, 2020. Curran Associates Inc

    2020

  8. [16]

    Leave no Trace: Learning to Reset for Safe and Autonomous Rein- forcement Learning

    Benjamin Eysenbach, Shixiang Gu, Julian Ibarz, and Sergey Levine. Leave no Trace: Learning to Reset for Safe and Autonomous Rein- forcement Learning. November 2017. arXiv:1711.06782 [cs]

    2017 arXiv

  9. [17]

    There is no turning back: A self-supervised approach for reversibility-aware reinforcement learning

    Nathan Grinsztajn, Johan Ferret, Olivier Pietquin, Philippe Preux, and Matthieu Geist. There is no turning back: A self-supervised approach for reversibility-aware reinforcement learning. In A. Beygelzimer, Y . Dauphin, P. Liang, and J. Wortman Vaughan, editors,Advances in Neu...

    2021

  10. [18]

    Dy- namic model predictive shielding for provably safe reinforcement learning

    Arko Banerjee, Kia Rahmani, Joydeep Biswas, and Isil Dillig. Dy- namic model predictive shielding for provably safe reinforcement learning. InThe Thirty-eighth Annual Conference on Neural Infor- mation Processing Systems, 2024

    2024

  11. [19]

    Rehg, and Evangelos A

    Grady Williams, Paul Drews, Brian Goldfain, James M. Rehg, and Evangelos A. Theodorou. Aggressive driving with model predictive path integral control. In2016 IEEE International Conference on Robotics and Automation (ICRA), pages 1433–1440, Stockholm, May

  12. [20]

    Gymnasium: A standard in- terface for reinforcement learning environments.arXiv preprint arXiv:2407.17032, 2024

    Mark Towers, Ariel Kwiatkowski, Jordan Terry, John U Balis, Gian- luca De Cola, Tristan Deleu, Manuel Goul ˜ao, Andreas Kallinteris, Markus Krimmel, Arjun KG, et al. Gymnasium: A standard in- terface for reinforcement learning environments.arXiv preprint arXiv:2407.17032, 2024

    2024 arXiv

  13. [21]

    Proximal policy optimization algorithms.ArXiv, abs/1707.06347, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.ArXiv, abs/1707.06347, 2017

    2017 arXiv

  14. [22]

    Stable-baselines3: Reliable reinforcement learning implementations.Journal of Machine Learning Research, 22(268):1–8, 2021

    Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Max- imilian Ernestus, and Noah Dormann. Stable-baselines3: Reliable reinforcement learning implementations.Journal of Machine Learning Research, 22(268):1–8, 2021

    2021

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.