{"id":"52e53fd2-c428-44ab-9cf5-479f6d2ce914","arxiv_id":"2507.22389","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"FORCE-OPT extracts calibrated, multi-modal reachable sets from GMM trajectory predictors using convex optimization and conformal prediction, achieving the lowest balanced error rate in safety evaluation on nuScenes.","lead":"The paper presents FORCE-OPT, a safety monitor that learns forward reachable sets from trajectory predictors to check whether a self-driving motion plan collides with other agents. On nuScenes, FORCE-OPT achieves the best balance of false positives and false negatives among the baselines, at inference times under 25 milliseconds.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported ID coverage (89.95%) is far below the 95% conformal target and the calibration split is not disclosed as held-out; if calibration overlaps the training split, the coverage guarantee is void and the central completeness claim collapses.","rationale":"The paper's strongest claim is that FORCE-OPT achieves the best soundness-completeness balance. The computational machinery (convex extraction, covariance-scaling invariance, conformal calibration) is coherent: Theorem 2 and Corollary 1 are plausible, and the non-conformity score has a closed form. The load-bearing element for the safety guarantee is the split-conformal step in Section IV-C, because it converts a learned, uncalibrated GMM into a set with a stated coverage probability. That step is only valid if calibration scores are exchangeable with test scores. The paper does not state whether the 35,220 calibration scenes are disjoint from the Singapore training split used to train Autobots, and the reported ID coverage (89.95%) is far below the calibrated 95% target, a gap much larger than finite-sample slack. This is a concrete red flag that the guarantee is not being realized in the reported experiments. The FNR results in Tables I and II therefore cannot be taken as evidence of deployment-time completeness. Secondary concerns (no error bars, hand-picked beta thresholds, ambiguous trajectory-level Cov metric) reinforce the conditional verdict, but the calibration-split issue is the most direct threat. A single check, disclosing the calibration split and reporting per-state miscoverage on a fresh held-out test split, would settle whether the central claim's safety guarantee actually holds.","tokens_in":14178,"tokens_out":19967,"duration_ms":229607,"concrete_test":"Ask the authors to disclose the provenance of the 35,220 calibration scenes (training split vs. held-out split). With FORCE-OPT fixed, compute the empirical miscoverage rate on a fresh test split disjoint from calibration, i.e., the fraction of ground-truth agent states outside C(s, eta). If the calibration split is disjoint and the rate exceeds 0.05 + sqrt(log(1/delta)/(2N)) approximately 0.056 for N=35,220, the conformal guarantee fails; if calibration used training data, re-calibrate on a held-out split and recompute Tables I-II.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The safety guarantee rests on the split-conformal calibration in Section IV-C: the FRS is claimed to cover ground-truth trajectories with high probability. This requires the non-conformity scores of the calibration scenes to be exchangeable with test scores. The paper trains Autobots on the Singapore training split (Section V-A), then states that 'We use a dataset with a cardinality of 35220 for calibration' (Section V-D) without saying whether those calibration scenes are disjoint from the training split. If they come from the training split, the predictor's in-sample scores are overconfident, so the calibrated scaling factor eta is too small and the guarantee breaks on fresh data. A direct red flag appears in Table I: FORCE-OPT's reported ID coverage is 89.95%, five percentage points below the 95% target. With N=35,220, the finite-sample correction sqrt(log(1/delta)/(2N)) is below 0.7%, so this gap is far too large to be explained by calibration randomness. Thus either the calibration set is not exchangeable with the evaluation set, or the reported 'Cov' metric is not the conformal coverage object being claimed. In both cases, the paper's central completeness guarantee is not established by the reported experiments, and the FNR of 3.03% cannot be taken as a reliable deployment-time miss rate.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FORCE-OPT, a runtime safety monitor for motion plans. The method treats a multi-modal GMM trajectory predictor as a stochastic model of other agents and extracts a forward reachable set (FRS) by solving a convex program that minimizes the total volume of Gaussian sublevel sets subject to a probability-mass constraint. Split conformal prediction is then used to calibrate the FRS by scaling the GMM covariances, with the non-conformity score defined as the minimal scaling factor that covers the ground-truth trajectory. To handle distribution shift, the authors add a Bayesian filter that adaptively dilates the covariance based on the predictor's observed likelihood, with fallback to parameterized or worst-case reachability when the estimated confidence drops below a threshold. The method is evaluated on nuScenes, using an Autobots predictor trained on Singapore and tested on both Singapore (in-distribution) and Boston (out-of-distribution), with synthetic unsafe scenarios to measure false negatives. The main empirical claim is that FORCE-OPT achieves the best balance of false-positive and false-negative rates, with a BER of 5.83% in-distribution and 6.77% out-of-distribution for the best variant, at roughly 22 ms per safety check.","tokens_in":14503,"tokens_out":11240,"duration_ms":144943,"significance":"If validated, the paper offers a practical and principled route from learned trajectory predictors to calibrated reachability-style safety monitors. The convex formulation in Theorem 2 is correctly derived, and Corollary 1 (invariance of the optimizer under covariance scaling) is true, which makes the conformal non-conformity score analytically tractable. The use of a public dataset, a public predictor, and a synthetic unsafe-scenario construction is a strength; the reported runtime is competitive for online monitoring. The core idea---calibrating FRSs extracted from GMM predictors rather than using raw or worst-case sets---is timely and likely useful for learned autonomy stacks. However, the conformal guarantee and the empirical coverage numbers are currently not reconciled, and the proof of the theoretical FRS equivalence in Theorem 1 has a gap. These issues bear directly on the central claims of completeness and soundness, so they need to be addressed before the contribution can be fully accepted.","major_comments":[{"comment":"The paper does not disclose whether the 35,220 calibration examples are disjoint from the nuScenes training split used to train Autobots. Split conformal prediction requires exchangeability of calibration and test non-conformity scores; if the calibration data overlap the training split, the scores are in-sample, the calibration factor eta is too small, and the guarantee in Eq. (8) does not transfer to fresh data. This concern is reinforced by Table I: FORCE-OPT's reported ID coverage is 89.95%, which is about five percentage points below the 95% target. With N=35,220 the finite-sample correction term is below 0.7%, so the gap is far too large to be explained by calibration randomness. The authors should disclose the exact split, define precisely what the Cov metric counts (per scene, per agent, or per agent-timestep), and state which empirical quantity corresponds to the conformal coverage object in Eq. (8).","section":"V-D and Table I"},{"comment":"The proof of Theorem 1 is not valid as written. It invokes Luzin's N-property to claim that a measurable subset F of F_t with positive volume has a preimage of positive Lebesgue measure under the composed dynamics map. Luzin's N-property concerns forward images of null sets and does not imply this; if the domain of the composed map has lower dimension than X, the image has measure zero, and even in equal dimension the rank conditions are nontrivial. The proof also assumes that the absolutely continuous measure on the control/disturbance space assigns positive probability to every positive-measure subset of its support, which is not stated. Without these assumptions, the support of mu_t can be a proper subset of F_t, in which case the minimal volume set omega*_t need not equal F_t up to a measure-zero set. The authors should either add the required assumptions or weaken Theorem 1 to a statement about the support of the pushforward measure.","section":"Appendix, Proof of Theorem 1"},{"comment":"Equation (8) as printed is not the standard split-conformal finite-sample bound. With delta in (0,1), the term sqrt(-log delta / 2N) is positive, so the displayed inequality states that the miscoverage probability is less than gamma minus a positive term. The standard form from [39, Proposition 2a] is gamma plus sqrt(log(1/delta)/(2N)) (equivalently, with the appropriate quantile-index ceiling). Please correct the sign and state the exact quantile construction used for eta. This matters because the current text presents a formal guarantee that is both nonstandard and inconsistent with the observed 10.05% miscoverage rate in Table I.","section":"IV-C, Eq. (8)"},{"comment":"All performance numbers are reported as single-point estimates with no confidence intervals, error bars, or number of calibration runs. Because split conformal calibration introduces randomness (and the predictor training may as well), the claimed ordering---e.g., FORCE-OPT's ID BER of 5.83% versus 7.78% for FORCE-OPT + belief, or the OOD ordering among FORCE-OPT variants in Table II---may be within noise. The authors should report means and standard deviations over multiple calibration splits, and provide a sensitivity analysis for the hand-picked Bayesian thresholds (beta_low=0.3, beta_high=1, switch at beta<0.75) since the OOD conclusions depend on those choices.","section":"Tables I-II and V-E"}],"minor_comments":[{"comment":"The notation \\bar{x}_t in Eq. (13) is not defined; presumably it denotes the mean of the t-th GMM component, but it should be introduced explicitly.","section":"IV-D, Eq. (13)"},{"comment":"The proof by contradiction assumes that c*_{i,alpha1} < c*_{i,alpha2} for a single index i and then compares the full sums as if the inequality held componentwise. The invariance statement is true because the covariance scaling factors out of the objective and leaves the constraint unchanged, but the proof as written is not rigorous and should be rewritten.","section":"Appendix, Proof of Corollary 1"},{"comment":"The description of the Nakamura et al. baseline contains a typo ('3% probaility mass') and is ambiguous; it should state clearly whether the control bounds enclose the 99% or 3-sigma region.","section":"V-D"},{"comment":"The first sentence of the conclusion contains a stray footnote marker '4' that appears to be a formatting artifact.","section":"VI"},{"comment":"The caption refers to 'true future positions of the contender (the cyan ellipses do not cover the true ground truth states states shown with red dots)' and later to 'containing the agent's true future path (green dots)'; the color convention for ground truth in panels (b) and (c) is inconsistent and should be reconciled.","section":"Fig. 1 caption"},{"comment":"The value of delta in the conformal guarantee of Eq. (8) is never specified in the experiments; please state the chosen delta and report the resulting finite-sample correction.","section":"IV-C"}],"recommendation":"major_revision","confidential_remarks":"The paper's core method is sound in conception: the convex FRS extraction is efficient, the conformal calibration idea is appropriate, and the empirical setup is on a public dataset. The main blockers are fixable: the calibration split and coverage metric must be disclosed, the conformal bound in Eq. (8) must be corrected, and Theorem 1 needs either a repaired proof or a weakened statement. The comparison to Nakamura et al. [25] is not circular in my view, since [25] is published prior work and the Bayesian update is not the paper's target contribution. I recommend major revision rather than rejection; the fixes are within the scope of a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper is worth a careful look. FORCE-OPT does something new: it turns the GMM output of a trajectory predictor into a forward reachable set by solving a small convex program, then uses conformal prediction to scale the GMM covariances to get a coverage guarantee. The invariance corollary (the optimal sub-level set thresholds don't change under covariance scaling) is elegant and makes the non-conformity score analytic. The multimodality ablation--more modes reduce the needed inflation--is a non-obvious and informative result. Runtime around 22 ms is genuinely practical, and the BER numbers beat the baselines on nuScenes.\n\nThe soft spot is the empirical validation of the conformal guarantee. Table I reports an ID coverage of 89.95% for FORCE-OPT, five points below the 95% target. With N=35,220, the finite-sample bound is under 0.7%, so that gap is far too large to be calibration randomness. Either the calibration set overlaps the training split (the paper never says it is held out) or the Cov metric is not the conformal coverage object. This matters because the completeness claim rests entirely on that guarantee. The authors need to disclose the calibration split and define Cov precisely.\n\nMinor issues: no error bars or significance tests; the synthetic unsafe set for FNR is not described in size, so those numbers may be noisy; the Bayesian filter thresholds (beta_low=0.3, beta_high=1, switch at 0.75) are ad hoc and carry no formal guarantee, though they are reasonable as a heuristic. The Theorem 1 proof is terse but the argument seems sound; I wouldn't call it a gap. The abstract overstates \"complete\" when the actual guarantee is probabilistic.\n\nOverall, the method is a solid contribution and deserves a serious referee. The coverage inconsistency is a load-bearing issue but likely fixable with a clearer experimental section or a re-calibration. I'd send it to review.","headline":"Useful new combination of convex FRS extraction and conformal calibration; the experimental coverage gap needs a clear explanation before the completeness claim can be trusted.","tokens_in":15018,"tokens_out":6002,"would_cite":false,"duration_ms":69516,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FORCE-OPT extracts calibrated forward reachable sets from Gaussian-mixture trajectory predictors via convex optimization and conformal prediction, adapts them with a Bayesian filter out-of-distribution, and achieves the best balance of…","keywords":["forward reachable sets","trajectory prediction","conformal prediction","safety monitoring","autonomous driving","Gaussian mixture models","convex optimization","out-of-distribution robustness"],"falsifier":"Measure the empirical coverage of FORCE-OPT's reachable sets on a test set with a distribution shift of a different kind than city-level, such as night-time or rainy driving from a different sensor setup; if the empirical coverage falls below the nominal $1-\\gamma$ level while the Bayesian belief stays high, then the completeness guarantee fails in exactly the regime the filter was meant to handle.","tokens_in":14016,"feed_emoji":"🚗","tokens_out":7500,"duration_ms":72178,"temperature":0.7,"pith_summary":"This paper aims to show that modern trajectory predictors—neural networks trained to forecast where other agents will go—can serve as forward reachable set estimators for safety monitoring, giving tighter sets than worst-case reachability while keeping the monitor complete. The proposed FORCE-OPT algorithm extracts a minimal-volume reachable set from the predictor's Gaussian mixture output by solving a convex program, calibrates that set with conformal prediction so it covers the ground-truth future with a user-set probability, and adds a Bayesian filter that dilates the set when the predictor appears unreliable under distribution shift. On nuScenes, FORCE-OPT reports the lowest balanced error rate—the mean of false positives and false negatives—among calibrated, uncalibrated, and worst-case baselines, both in-distribution and when tested on a different city, with an inference time of about 22 ms per frame. The paper also proves that the probabilistic forward reachable set it optimizes is equivalent, almost everywhere, to the classical worst-case forward reachable set.","feed_headline":"Calibrated reachable sets cut safety-monitor errors to 5.8% on nuScenes","feed_subtitle":"A convex program plus conformal prediction turns learned forecasts into calibrated safety sets at 22 ms per check.","key_machinery":"The load-bearing object is the calibrated FRS constructed from the GMM modes of a trajectory predictor. For each mode $i$, FORCE-OPT solves the convex program $\\min_{c_i} \\sum_i \\pi\\sqrt{\\lambda_{i,1}\\lambda_{i,2}}\\, c_i$ subject to $\\sum_i p_i(1-e^{-c_i/2}) \\ge \\tau$, where the $c_i$ are Mahalanobis sublevel-set radii and $\\lambda_{i,1},\\lambda_{i,2}$ are the eigenvalues of each component's covariance; the resulting union of ellipses is the reachable set. Conformal prediction then rescales every covariance by a common factor $\\eta$ chosen from the empirical quantile of the non-conformity score $\\psi(s,x)=\\min_i (V_i(x)/c_i)$, which the invariance property of the convex program makes analytic. A Bayesian filter over a two-point belief adjusts this set further by scaling covariances by $1/\\hat{\\beta}$ when the predictor's observed performance degrades.","core_discovery":"The central discovery is that the probabilistic forward reachable set—the smallest set containing the full probability mass under the agent's state distribution—coincides almost everywhere with the classical worst-case reachable set, which justifies estimating reachable sets from learned predictors. FORCE-OPT turns this into a tractable algorithm: given a GMM trajectory predictor, it solves a convex optimization over the sublevel-set radii of each Gaussian mode to minimize total ellipsoid volume subject to capturing at least a fraction $\\tau$ of the probability mass, then applies split conformal prediction by scaling all covariances by a single factor $\\eta$, chosen from a calibration set so that the set covers the true future with high probability. The covariance-scaling invariance of the convex program lets the conformal factor be computed analytically from a non-conformity score per scene. On nuScenes, this yields the best balance of false positive and false negative rates across all baselines—5.83% balanced error in-distribution and 9.09% out-of-distribution, dropping to 6.77% with the belief-based worst-case fallback.","pith_inferences":["The same convex extraction could apply to other mixture density estimators besides Gaussians—student-$t$ or normalizing-flow mixtures—by replacing the $\\chi^2$ tail in the constraint with the appropriate distribution function, though the analytic conformal-score form would need re-deriving.","The two-point Bayesian belief over 'low' and 'high' confidence values is a coarse hedge; a deployment-grade monitor should also track the running empirical coverage of the sets and flag when it deviates from the nominal level, since the conformal guarantee no longer holds once the exchangeability assumption fails.","Because the belief thresholds are hand-picked, the Bayesian adaptation could be made testable by learning the belief-transition parameters from logged predictor failures, and one concrete experiment would be measuring whether the resulting balanced error improves over the hand-tuned values."],"forward_implications":["FORCE-OPT achieves a balanced error rate of 5.83% in-distribution and 9.09% under city-level out-of-distribution shift, lower than every baseline tested, while running in about 22 ms per frame.","Adding the belief-based fallback (FORCE-OPT + pWC-FRS) lowers the out-of-distribution balanced error to 6.77%, showing graceful degradation rather than a hard failure when the predictor becomes unreliable.","Increasing the number of GMM modes reduces the conformal inflation factor and improves the false-positive rate, so multi-modality in the predictor translates directly into a less conservative safety monitor.","Uncalibrated predictors miss a large share of genuine collisions (false-negative rates up to 55.56% out-of-distribution), while worst-case reachability sets trigger false alarms on roughly 44% of safe scenes, motivating calibration as the middle path."],"supporting_citations":[{"why":"Supplies the multi-modal GMM trajectory predictor whose output distributions FORCE-OPT converts into reachable sets.","marker":"[4]"},{"why":"Provides the split conformal prediction framework used to calibrate the FRS coverage.","marker":"[5]"},{"why":"Supplies the conditional validity result (Proposition 2a) that yields the high-probability coverage bound in (8).","marker":"[39]"},{"why":"Supplies the Bayesian belief-update scheme for tracking predictor confidence and serves as a baseline.","marker":"[25]"},{"why":"A conformal-prediction safety monitor baseline that FORCE-OPT compares against and outperforms in balanced error.","marker":"[26]"},{"why":"The nuScenes dataset used to train the predictor, build calibration sets, and evaluate in-distribution and out-of-distribution performance.","marker":"[7]"},{"why":"Prior probabilistic reachable-set formulation that motivates Theorem 1's equivalence result.","marker":"[32]"},{"why":"A multi-modal conformal region method that requires sampling, contrasted with FORCE-OPT's sample-free convex extraction.","marker":"[34]"},{"why":"Gives the volume formula for n-dimensional ellipsoids used to write the objective of the convex program (6).","marker":"[40]"},{"why":"Gives the chi-squared distribution of quadratic forms of Gaussian vectors used to express the probability-mass constraint in (6).","marker":"[41]"}],"fun_headline_variants":["Safety monitor uses predictors as reachable sets, 5.8% error on nuScenes","Predictor-based safety monitor: 5.8% error, 22 ms checks","Convex+conformal reachable sets slash safety monitor false alarms","Safety monitor: data-driven reachable sets with worst-case fallback","22 ms per check, 5.8% error: new safety monitor for autonomy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conformal coverage guarantee assumes test scenes are exchangeable with the calibration set, and the Bayesian filter's hand-picked thresholds have no formal coverage guarantee, so if the predictor's error pattern shifts in ways the belief model does not capture, the low false-negative rate can degrade.","fun_headline_variants_meta":{"raw":{"variants":["Safety monitor uses predictors as reachable sets, 5.8% error on nuScenes","Predictor-based safety monitor: 5.8% error, 22 ms checks","Convex+conformal reachable sets slash safety monitor false alarms","Safety monitor: data-driven reachable sets with worst-case fallback","22 ms per check, 5.8% error: new safety monitor for autonomy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000666,"raw_usage":{"total_tokens":3081,"prompt_tokens":1027,"completion_tokens":2054,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":643,"completion_tokens_details":{"reasoning_tokens":1949}},"tokens_in":643,"tokens_out":2054,"duration_ms":15548,"temperature":1.0,"reasoning_tokens":1949,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T11:45:06.107994+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the empirical coverage of FORCE-OPT's reachable sets on a test set with a distribution shift of a different kind than city-level, such as night-time or rainy driving from a different sensor setup; if the empirical coverage falls below the nominal $1-\\gamma$ level while the Bayesian belief stays high, then the completeness guarantee fails in exactly the regime the filter was meant to handle.","supporting_citations":[{"cited_title":"Conditional validity of inductive conformal predictors,","cited_arxiv_id":null,"evidence_quote":"Supplies the conditional validity result (Proposition 2a) that yields the high-probability coverage bound in (8)."},{"cited_title":"Online update of safety assurances using confidence-based predictions,","cited_arxiv_id":null,"evidence_quote":"Supplies the Bayesian belief-update scheme for tracking predictor confidence and serves as a baseline."},{"cited_title":"Safe planning in dynamic environments using conformal prediction,","cited_arxiv_id":null,"evidence_quote":"A conformal-prediction safety monitor baseline that FORCE-OPT compares against and outperforms in balanced error."},{"cited_title":"nuscenes: A multimodal dataset for autonomous driving,","cited_arxiv_id":null,"evidence_quote":"The nuScenes dataset used to train the predictor, build calibration sets, and evaluate in-distribution and out-of-distribution performance."},{"cited_title":"Robust, Informative Human-in-the-Loop Predictions via Empirical Reachable Sets","cited_arxiv_id":"1705.00748","evidence_quote":"Prior probabilistic reachable-set formulation that motivates Theorem 1's equivalence result."},{"cited_title":"Multi-modal conformal prediction regions by optimiz- ing convex shape templates,","cited_arxiv_id":null,"evidence_quote":"A multi-modal conformal region method that requires sampling, contrasted with FORCE-OPT's sample-free convex extraction."},{"cited_title":"V olume of n-dimensional ellipsoid,","cited_arxiv_id":null,"evidence_quote":"Gives the volume formula for n-dimensional ellipsoids used to write the objective of the convex program (6)."},{"cited_title":"The distribution of quadratic forms of gaussian vectors,","cited_arxiv_id":null,"evidence_quote":"Gives the chi-squared distribution of quadratic forms of Gaussian vectors used to express the probability-mass constraint in (6)."}],"review_version":1}