{"id":"21f083d6-3564-4d86-a744-03171de15a40","arxiv_id":"2506.17927","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"The authors propose a probability-space safety certificate for latent-variable systems, expressed through a marginalized Q-function, with a persistent-feasibility guarantee and a connection to causal reinforcement learning.","lead":"This paper builds safety certificates for control systems where hidden variables shift the statistics between the data used for training and the data seen online. The certificate filters actions using a long-term risk function that can, in principle, be learned from offline data with causal reinforcement learning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Assumption 2.1 restricts the certificate to latent variables that are conditionally independent of past states, actions, and U_t given X_t; persistent latent states (including the paper's motivating examples) are outside the proved guarantee.","rationale":"The reader's weakest-assumption diagnosis is correct and is the most load-bearing concern. Under Assumption 2.1, the observable state is Markov, equations (3)-(4) are well-defined, and the value/Q-function representation in Proposition 3.1 and Lemma 3.3 is sound. The induction in Theorem 3.2 and the feasibility argument in Theorem 3.4 are also internally coherent, and the paper deserves credit for a genuinely novel integration of causal RL with probabilistic safety certificates. However, the same assumption is the linchpin of the entire architecture, and it is much more restrictive than the paper's broad language about latent variables suggests. Persistent or evolving latent states, which are natural in the occlusion and road-slipperiness examples, violate the assumption, and then no function of the current observable state and remaining time can carry the information needed for the certificate. Because this is a structural scope limitation rather than an algebraic error, it does not invalidate the conditional contribution, but it should be explicitly rescoped or tested. The proposed experiment directly probes whether persistent latent variables break the guarantee; if it fails, the paper should be revised to state that the certificate applies only when Assumption 2.1 holds and to discuss the resulting restriction prominently. This keeps the reader's conditional verdict unchanged.","tokens_in":17944,"tokens_out":22083,"duration_ms":232896,"concrete_test":"Modify the §4 driving simulation so that W_t is persistent: replace the i.i.d. draws from Appendix C with a Markov chain satisfying P(W_{t+1}=W_t|W_t)=0.9, whose stationary distribution at each road-position class equals the marginal P(W_t|X_t) used there. Keep all other parameters, the oracle-Q setup, Algorithm 2, and the 100x100 trajectory protocol unchanged. Compute the empirical closed-loop long-term safe probability P_{\\hat\\pi,\\hat\\pi}(C(X_0)\\cap\\cdots\\cap C(X_H)|X_0). If it falls below 1-\\epsilon = 0.8, the certificate's guarantee depends on Assumption 2.1 and does not extend to the persistent-latent regime the paper's motivation invokes.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The entire construction rests on the observed process X_t being Markov. Assumption 2.1 is what makes equation (4) a valid one-step online transition and makes the long-term safe probability in (7) a function of X_t alone. Without it, the marginalized process P(X_{t+1}|X_t,U_t) is not a sufficient statistic for future risk, and the safety condition S(X_t,U_t,t) in (27), which depends only on the current observable state and remaining time, cannot encode the latent variable's influence on the future. Assumption 2.1 is far stronger than 'latent variables exist': it requires W_t to be independent of all past latent variables, past observable states, and all actions up to and including U_t, given X_t. This rules out persistent latent states (W_{t+1} depending on W_t), hidden road-slipperiness that evolves in time, and pedestrian intent that persists across frames, even though these are exactly the motivating scenarios. The assumption is stated in the problem formulation but is not listed as a limitation and is absent from the abstract's scope statement. There is also a tension with the offline behavioral policy: Section 2.1 defines offline data via pi_b(U_t|X_t,W_t), which makes U_t depend on W_t given X_t, violating the stated W_t independence from U_t unless the assumption is reinterpreted as an interventional condition in the online system only; this needs clarification. The proofs of Theorem 3.2 and Theorem 3.4 are internally consistent given an exact Q-function and Assumption 2.1, so the concern is not about the algebra but about the reach of the central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a probabilistic safety certificate for stochastic control systems with latent variables that induce a distribution shift between offline and online observed statistics. Under a conditional-independence assumption on the latent process (Assumption 2.1), the authors define a marginalized value function V^pi and Q-function Q^pi on an auxiliary MDP with an absorbing unsafe state, and show that the condition S(X_t,U_t,t) = Q^pi(Yhat_t,U_t) - E_{U~pi}[Q^pi(Yhat_t,U)|Yhat_t] >= 0 is sufficient for the long-term safe probability to remain above 1-epsilon (Theorem 3.2, Lemma 3.3). They also prove that a feasible action always exists (Theorem 3.4) and sketch an offline estimation procedure based on front-door adjustment and iterative Q-estimation (Algorithm 2, Section 3.4). The method is illustrated on a discrete driving simulation.","tokens_in":18187,"tokens_out":11587,"duration_ms":114022,"significance":"The conceptual bridge between probability-space invariance conditions and marginalized Q-functions is attractive and could be useful: if an exact or consistently estimated Q^pi is available, the resulting safety filter is simple, online, and comes with a persistent-feasibility guarantee. The proofs of Theorems 3.2 and 3.4 are elementary and correct given the paper's assumptions. The paper also correctly identifies that offline statistics alone are unreliable under latent-variable confounding, and it draws on a credible causal-RL literature to justify estimating Q^pi from observed data. However, the significance is conditional: the central assumption as written conflicts with the paper's own problem setup, and the simulation does not exercise the proposed learning pipeline, so the claim that the certificate 'can be constructed using observed statistics' is not yet demonstrated.","major_comments":[{"comment":"Assumption 2.1 states W_t is independent of {U_tau}_{tau<=t} given X_t. Under this assumption, the offline behavioral policy pi_b(U_t|X_t,W_t) must satisfy pi_b(U_t|X_t,W_t) = pi_b(U_t|X_t), because U_t is conditionally independent of W_t given X_t. Then Eq. (3) reduces to Eq. (4), and the offline/online distribution shift that motivates the paper disappears. The simulation's behavioral policy given in Appendix C explicitly depends on W_t given X_t, so the simulated setting violates Assumption 2.1 as written. This is load-bearing: the mismatch between Eqs. (3) and (4) is the foundation for the paper's problem statement. Please clarify whether Assumption 2.1 is intended to hold only for the online system, or remove U_t from the independence statement, and adjust the derivations accordingly.","section":"Section 2.1, Assumption 2.1 and Eqs. (3)-(4)"},{"comment":"The numerical study does not implement the proposed Algorithm 2. The text states that the proposed method 'has access to an unbiased estimate for the Q-function Q^pi', which can be estimated by an existing causal RL method, but the simulation never runs the iterative procedure in Eq. (44), never uses the offline dataset D to estimate Q^pi_M, and never introduces the mediator M_t required by Assumption 3.5. Thus the central claim that the safety certificate can be constructed from observed statistics is not empirically validated. Please provide an end-to-end experiment that learns Q^pi from D, or at minimum a sensitivity study showing how the guarantee degrades with estimation error.","section":"Section 4, Simulation"},{"comment":"Assumption 2.1 rules out persistent latent states: W_t is independent of all past latent variables and past observable states given X_t, so W_{t+1} cannot depend on W_t, and a latent variable that evolves over time (e.g., a slowly changing road-slipperiness parameter or a persistent pedestrian intent) is outside the proved guarantee. These are exactly the motivating examples in the Introduction. This limitation is not disclosed in the abstract or in a limitations paragraph, and the title's unqualified reference to 'latent variables' overstates the scope. Please either narrow the claims to conditionally independent latent variables or add a prominent statement of the limitation.","section":"Section 2.1 and Introduction"}],"minor_comments":[{"comment":"The displayed decomposition P(X_{t+1},W_{t+1}|X_t,W_t,U_t) = P(W_t|X_t)P(X_{t+1}|X_t,U_t,W_t) is dimensionally inconsistent: the left-hand side conditions on W_t, so the right-hand side should not contain the marginal P(W_t|X_t). Please correct this to a proper transition kernel for W_{t+1}, or state the intended factorization more carefully.","section":"Section 2.1, decomposition after Assumption 2.1"},{"comment":"Assumption 3.5 says that the mediator M_t intercepts every directed path 'from U_t to U_t or to S_{t+1}', but S_{t+1} is never defined and the path 'from U_t to U_t' appears to be a typo. Please rewrite the assumption in terms of the state transition and define all variables.","section":"Section 3.4, Assumption 3.5"},{"comment":"The algorithm says 'Estimate Q^pi using (54) with Q^pi_M', but Eq. (54) also requires the offline conditional distributions P_offline(M_t|U_t,Yhat_t). It should be stated explicitly that these distributions are estimated from the dataset D and how the estimation error is handled.","section":"Algorithm 2, line 14"},{"comment":"The simulation section reports 95% confidence intervals but does not describe how they are computed (e.g., whether they are per-time binomial intervals or across trajectory replications). Please add a brief description of the statistical procedure.","section":"Section 4, Figures 1-2"}],"recommendation":"major_revision","confidential_remarks":"The core idea is promising, and the proofs are correct under a clarified set of assumptions. The main obstacle is not mathematical in the narrow sense but conceptual: Assumption 2.1 as written conflicts with the existence of a confounded offline behavioral policy and with the simulation setup. If the authors can cleanly separate the online conditional-independence assumption from the offline logging-policy behavior, the paper could be salvageable with a reworked problem statement. The simulation gap is also substantial: an oracle-Q experiment does not support the paper's headline claim of constructing certificates from observed statistics. I would not accept the paper in its current form, but I see a path to revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper has a clean theoretical core, but the experimental section doesn't exercise the proposed learning pipeline, and the load-bearing assumption is narrower than the abstract suggests. I'd send it to review, but it needs major revision.\n\nThe genuinely new thing is the integration: a probability-space safety certificate built on a marginalized Q-function, plus a front-door adjustment (Shi et al. 2024) to estimate that Q-function from confounded offline data. Given an exact Qπ and Assumption 2.1, the invariance argument in Theorem 3.2, Lemma 3.3, and Theorem 3.4 is correct, and the persistent-feasibility guarantee is a nice touch. The writing is clear about the causal machinery. This fills a real gap: most safety certificates assume full state observations or known dynamics.\n\nThe soft spots are proportionate to the claims. First, Assumption 2.1 is doing a lot more work than the paper admits. It requires W_t to be independent of all past states, past latent variables, and all actions up to and including U_t, given X_t. That rules out persistent latent states—pedestrian intent, evolving road slipperiness—even though those are the motivating examples. The stress-test note is right, and it is a genuine problem that this restriction appears only in the problem formulation, not in the abstract or the stated scope. Second, the problem statement has an internal tension: offline actions are generated by π_b(U_t|X_t,W_t), which makes U_t depend on W_t given X_t, violating Assumption 2.1 as written. The paper needs to clarify that the assumption applies to the online system only, or the offline statistics in equation (3) are not compatible with it. Third, the simulation does not actually run the proposed causal RL estimator; it hands the controller an oracle Q-function. The comparison against DTCBF also gives the baseline less information. So the end-to-end claim—constructing the certificate from observed statistics—is not demonstrated. Fourth, there is no analysis of how estimation error in Qπ propagates into the safety guarantee. The theorems are exact-Qπ results; a finite-sample or robust version is needed to justify the method.\n\nMinor: the displayed decomposition of P(X_{t+1},W_{t+1}|...) in Section 2.1 looks typo'd, since W_{t+1} is missing from the right-hand side. It doesn't affect the proofs, but it should be fixed.\n\nThis paper is for researchers working at the intersection of safe control and causal RL. The theoretical connection is worth publishing, but I would not cite it yet as a validated method. I'd recommend peer review with major revision: either implement the learning pipeline in the experiments, or explicitly rescope the contribution to 'safety certificate design given an accurate Q-function' and analyze estimation error.","headline":"Sound theoretical core, but the learning pipeline is never exercised and the load-bearing assumption is narrower than advertised.","tokens_in":18803,"tokens_out":5600,"would_cite":false,"duration_ms":51102,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"One Q-function can certify long-term safety under hidden variables","keywords":["safety certificate","latent variables","probabilistic invariance","causal reinforcement learning","distribution shift","long-term safety","confounded Markov decision process","marginalized Q-function"],"falsifier":"Construct a two-state system with a latent variable that persists across time, e.g. $W_t$ following a Markov chain on $\\{0,1\\}$ with $P(W_{t+1}=1\\mid W_t=1, X_t)=0.9$, violating Assumption 2.1, while keeping the same safe event and offline dataset structure as the Appendix A example. Train a causal RL estimator on offline data generated under a behavioral policy, enforce condition (27) at every step, and measure the empirical long-term safe probability over many online episodes; if it drops below $1-\\epsilon$ while (27) was satisfied throughout, then the certificate's guarantee relies essentially on the conditional-independence assumption rather than on the Q-condition alone.","tokens_in":17674,"feed_emoji":"🛡️","tokens_out":8991,"duration_ms":78555,"temperature":0.7,"pith_summary":"Systems whose latent variables make their dynamics partially unidentifiable, and whose offline training data statistically differ from online operation, cannot be certified by standard control barrier functions or forward-invariance conditions. This paper proposes a probabilistic safety certificate that instead works in probability space: it states a condition on a marginalized Q-function that can be evaluated from observed statistics. If the condition is enforced at every step, the probability of staying safe for the whole episode stays above $1-\\epsilon$, and a feasible safe action provably always exists. The construction links safety certificates to causal reinforcement learning, so existing off-policy estimators can supply the Q-function without ever observing the latent variable. Numerical simulations on a slippery-road driving example show the certificate meeting the bound while a discrete-time control barrier function fails under the same distribution shift.","feed_headline":"One Q-function can certify long-term safety under hidden variables","feed_subtitle":"Offline data, learned Q-function, and a per-step check keep the whole-episode safe probability above 1−ε.","key_machinery":"The load-bearing object is the marginalized Q-function $Q^\\pi$ defined on the augmented state $\\hat{Y}_t=[X_t^{\\top}, H-t]^{\\top}$, for the auxiliary MDP whose online transition kernel $\\tilde{P}_{\\text{online}}$ keeps the state frozen once the safe event $C(X_t)$ fails and otherwise follows the marginal online statistics $P_{\\text{online}}(X_{t+1}\\mid X_t, U_t)$. The identity that carries the argument is the Bellman relation $Q^\\pi(\\hat{Y}_t, U_t)=\\mathbb{E}[V^\\pi(\\hat{Y}_{t+1})\\mid \\hat{Y}_t, U_t]$ (the running reward is zero except at the horizon), which turns the certificate $S\\ge 0$ into the probability-space invariance condition $\\mathbb{E}[V^\\pi(\\hat{Y}_{t+1})\\mid \\hat{Y}_t, U_t]\\ge V^\\pi(\\hat{Y}_t)$. The front-door adjustment, a causal identity, supplies the bridge from offline data to online statistics: it expresses $\\tilde{P}_{\\text{online}}(\\hat{Y}_{t+1}\\mid \\hat{Y}_t, U_t)$ in terms of offline quantities using an observable mediator variable, which is what makes the Q-function learnable without access to the latent state.","core_discovery":"The central claim is that long-term safety, defined as $P_{\\hat{\\pi},\\pi}(C(X_t)\\cap\\cdots\\cap C(X_H)\\mid X_0)\\ge 1-\\epsilon$ for every $t$, is guaranteed by enforcing the per-step condition $S(X_t, U_t, t)=Q^\\pi(\\hat{Y}_t, U_t)-\\mathbb{E}_{U\\sim\\pi}[Q^\\pi(\\hat{Y}_t, U)\\mid\\hat{Y}_t]\\ge 0$, where $\\hat{Y}_t=[X_t^{\\top}, H-t]^{\\top}$ appends the remaining time to the observed state and $Q^\\pi$ is the marginalized Q-function of an auxiliary MDP whose transitions freeze once the safe event $C$ fails. Under this condition, Theorem 3.4 shows that a safe action always exists, so the certificate never forces the controller into an infeasible stall. The proof identifies the long-term safe probability with the auxiliary value function $V^\\pi$ (Proposition 3.1), converts the value-based invariance condition (17) into the Q-based condition (27) through a modified Bellman equation, and evaluates that condition using offline data via the front-door adjustment, which corrects for the confounding latent variable.","pith_inferences":["If latent variables persist over time given the observed state, the same safety score likely overestimates safety; augmenting the state with a belief or a sufficient statistic would be a natural extension.","The feasibility guarantee in Theorem 3.4 proves only existence, not robustness; when the Q-function estimate has high variance away from offline data coverage, adding a safety margin or a conservative threshold on $S$ would be prudent.","The score $S$ could double as a monitoring statistic: values near zero flag states close to irrecoverable, before the safety bound is actually violated.","Proposition 3.1 implies the remaining safety margin can be read off $V^\\pi$ at runtime, enabling headroom-aware action ranking rather than simple pass/fail filtering."],"forward_implications":["Enforcing the Q-function advantage condition at every step keeps the whole-episode safe probability above $1-\\epsilon$, and a feasible safe input always exists, so the certificate never dead-ends the controller.","Safety certification is decoupled from knowing the full transition dynamics: only a marginalized Q-function needs to be learned or estimated, and any causal reinforcement-learning estimator that respects the confounded Markov structure can supply it.","The construction allows a myopic, performance-oriented policy to be filtered by a safety layer without retraining, since the certificate is evaluated online using the observed state and remaining time.","Existing offline datasets collected under a behavioral policy that conditions on the latent variable can be reused: Algorithm 1 converts them into an auxiliary dataset with offline statistics, and the front-door adjustment recovers online statistics from them.","The approach extends naturally to unbounded disturbances, actuation limits, and time-varying safe sets encoded in the event $C(X_t)$."],"supporting_citations":[{"why":"Supplies the off-policy confidence-interval estimation method used in Algorithm 2 to learn the marginalized Q-function from confounded offline data.","marker":"Shi et al. 2024"},{"why":"Provides the front-door adjustment and the causal-path definitions used to express online statistics in terms of offline data.","marker":"Pearl, 2009"},{"why":"Offers a provably efficient causal reinforcement learning estimator for confounded MDPs, named as an alternative Q-function estimator compatible with the certificate.","marker":"Wang et al. 2021b"},{"why":"Introduces the value-function representation of maximal safety probability that the certificate adapts to the latent-variable setting.","marker":"Hoshino & Nakahira, 2024"},{"why":"Defines the discrete-time control barrier function baseline whose safety guarantee fails under the offline/online distribution shift in the simulations.","marker":"Cosner et al. 2023"},{"why":"Provides an earlier probabilistic safety-certificate formulation in probability space that this paper generalizes to latent variables.","marker":"Jing & Nakahira, 2022"}],"fun_headline_variants":["Q-function certifies safety despite hidden dynamics","Per-step Q-check guarantees whole-episode safety","One Q-function keeps long-term safe probability high","Causal RL powers safety certificates in latent-variable systems"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The guarantee depends on the latent variable at each time being conditionally independent of all past latent variables, observed states, and actions given the current observed state; if the hidden variable carries persistent or history-dependent influence, the safety certificate can no longer be derived.","fun_headline_variants_meta":{"raw":{"variants":["Q-function certifies safety despite hidden dynamics","Per-step Q-check guarantees whole-episode safety","One Q-function keeps long-term safe probability high","Causal RL powers safety certificates in latent-variable systems"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001532,"raw_usage":{"total_tokens":6169,"prompt_tokens":1022,"completion_tokens":5147,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":638,"completion_tokens_details":{"reasoning_tokens":5087}},"tokens_in":638,"tokens_out":5147,"duration_ms":35461,"temperature":1.0,"reasoning_tokens":5087,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:58:08.026335+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a two-state system with a latent variable that persists across time, e.g. $W_t$ following a Markov chain on $\\{0,1\\}$ with $P(W_{t+1}=1\\mid W_t=1, X_t)=0.9$, violating Assumption 2.1, while keeping the same safe event and offline dataset structure as the Appendix A example. Train a causal RL estimator on offline data generated under a behavioral policy, enforce condition (27) at every step, and measure the empirical long-term safe probability over many online episodes; if it drops below $1-\\epsilon$ while (27) was satisfied throughout, then the certificate's guarantee relies essentially on the conditional-independence assumption rather than on the Q-condition alone.","supporting_citations":[{"cited_title":"and Nakahira, Y","cited_arxiv_id":null,"evidence_quote":"Introduces the value-function representation of maximal safety probability that the certificate adapts to the latent-variable setting."},{"cited_title":"K., Culbertson, P., Taylor, A","cited_arxiv_id":null,"evidence_quote":"Defines the discrete-time control barrier function baseline whose safety guarantee fails under the offline/online distribution shift in the simulations."},{"cited_title":"and Nakahira, Y","cited_arxiv_id":null,"evidence_quote":"Provides an earlier probabilistic safety-certificate formulation in probability space that this paper generalizes to latent variables."}],"review_version":2}