{"id":"491c8a3c-96cd-4b5c-bb62-4231ba3370b2","arxiv_id":"2608.04246","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"SAFECAST augments hidden-state failure-probe training and conformal calibration with visual and language contrast sets, improving VLA failure detection under distribution shift in several tested settings.","lead":"This paper tests whether adding contrast set perturbations (visual clutter, reworded instructions) to the training and calibration data of a robot failure detector makes it better at spotting failures during deployment shifts. The method, SAFECAST, improves failure detection scores over the SAFE baseline in several simulated and real-robot settings, though some gains are not statistically significant and no code is released.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Real-world DROID evaluation may be confounded: D_eval perturbation families overlap with D_CS, so SAFECAST may be calibrating on eval-like data rather than demonstrating robustness to held-out deployment shift.","rationale":"The reader's weakest assumption was that contrast-set perturbations are representative of deployment shifts, which the paper itself concedes in Limitations. My concern sharpens this into a more specific internal-validity problem: in the real-world DROID experiments, D_eval may not be a held-out distribution at all, because the same perturbation families appear in D_CS and D_eval and no split disjointness is documented. If this concern lands, the headline claim that SAFECAST provides robust failure detection 'during distribution shifts' is not supported by the real-world experiments; if it does not land, the central empirical claim survives. This is a load-bearing concern because the real-world DROID results are the strongest evidence for the method, and unlike the LIBERO simulation results they are not backed by an independent pre-existing benchmark. I do not recommend changing the reader's CONDITIONAL verdict: the concern is serious but addressable with released data splits and a held-out-family analysis, and the simulation results plus extended ablations in Appendix F provide partial independent support. The abstract's statistical-significance overstatement for Sim-OpenVLA (p=0.0256, not significant after Bonferroni correction over 12 tests) is a real but secondary issue; it does not by itself undermine the central claim as much as the potential D_CS/D_eval overlap. My test would settle whether the real-world result is a genuine deployment-shift improvement or an artifact of overlapping perturbation pools.","tokens_in":25383,"tokens_out":6011,"duration_ms":59601,"concrete_test":"Run a leave-one-perturbation-family-out evaluation on the real DROID pool: train and calibrate SAFECAST on D_src plus contrast sets excluding one family (e.g., novel objects/tasks), then evaluate only on eval rollouts from that held-out family; repeat for each family (clutter, lighting, paraphrase, altered initial state). If the ROC-AUC gain over SAFE is not positive for held-out families, the reported advantage depends on calibrating on eval-like perturbations. Additionally, report instance-level overlap statistics between D_CS and D_eval, such as shared task IDs, object sets, and paraphrase templates, and a distributional distance (e.g., MMD) between probe risk trajectories from the two pools.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that D_eval be a genuinely held-out deployment distribution, not just a re-sampling of the same perturbation families used to build D_CS. In the real-world DROID experiments, Section 5.1 states that evaluation rollouts contain 'stronger shifts including novel objects, new tasks, clutter, lighting variation, and altered robot initial states.' Appendix A.2/Table 3 lists the contrast-set pool for the same robot as containing 'novel objects/tasks,' 'visual distractors/clutter,' and 'language paraphrases/negations.' The paper never states that D_eval is disjoint from D_CS, nor that different task instances, object sets, or paraphrase templates were used. If the same perturbation operators generated both pools, SAFECAST's augmented probe training and calibration data are drawn from the evaluation distribution, so the reported ROC-AUC gains over SAFE would largely reflect in-distribution calibration on eval-like rollouts rather than robustness to deployment shift. The LIBERO simulation experiments are less vulnerable to this concern because D_eval is the pre-existing LIBERO-Plus benchmark, but the real-world DROID results for both pi0 and pi0-FAST are a central pillar of the headline claim. The Limitations section acknowledges that performance may degrade outside the contrast-set distribution, but it does not address this more immediate split-overlap issue: the evaluation may not actually lie outside the contrast-set family in the real-world experiments.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"SAFECAST proposes to improve hidden-state failure detection for vision-language-action (VLA) policies under deployment-time distribution shift by augmenting both probe training and functional-conformal calibration with contrast-set rollouts (visual perturbations, language paraphrases, and their combination). The authors compare SAFE against three SAFECAST variants (train-augmentation only, calibration-augmentation only, and both) on LIBERO simulation with two base policies (π0 and OpenVLA) and on a real Franka/DROID setup with two checkpoints (π0 and π0-FAST), reporting F1 and ROC-AUC averaged over conformal significance levels α and 30 seeds. They report statistically significant improvements in most settings, a benefit of joint visual-language contrast sets, and a sim-to-real transfer result in which a simulation-trained probe calibrated on real contrast-set rollouts outperforms a real-only probe.","tokens_in":25625,"tokens_out":4873,"duration_ms":39900,"significance":"If the claims hold, SAFECAST is a practical, low-overhead intervention that strengthens runtime failure detection under distribution shift, which matters for safe deployment of VLA policies. The experimental methodology is a clear strength: 30 seeds, seed-paired t-tests, full alpha sweeps in the appendices, and controlled source/calibration/evaluation splits in simulation. The appendices also provide unusually thorough perturbation-family ablations and precision–recall diagnostics. However, the paper's headline significance claim is broader than the statistical evidence supports, and the real-world evaluation may be confounded by overlap between the contrast-set pool and the evaluation distribution. The significance of the work is therefore conditional on addressing these points.","major_comments":[{"comment":"The abstract states that SAFECAST 'statistically significantly improves' failure detection in both real-world DROID and LIBERO simulation experiments, but Table 6 shows that this is not true for all headline comparisons after the authors' own Bonferroni correction (m = 12, threshold 0.00417). In particular, Sim-OpenVLA SAFECAST vs. SAFE has p = 0.0256, Sim-OpenVLA SAFECAST_TRAINAUG has p = 0.4093, and Real-π0 SAFECAST_TRAINAUG has p = 0.8753. Please qualify the significance claim to the specific settings where it holds, or present a corrected multiple-testing analysis that justifies the abstract's wording.","section":"Abstract; §5.2; Table 6"},{"comment":"The real-world DROID evaluation may be confounded because the evaluation distribution D_eval overlaps with the contrast-set pool D_CS. Section 5.1 says evaluation rollouts contain 'stronger shifts including novel objects, new tasks, clutter, lighting variation, and altered robot initial states,' and Table 3 lists real-robot contrast-set perturbations including 'Visual distractors/clutter,' 'Language paraphrases/negations,' and 'Novel objects/tasks.' The paper never states that different task instances, object sets, or paraphrase templates were used in D_eval versus D_CS, nor that the perturbation operators were applied disjointly. If D_eval and D_CS are drawn from the same perturbation families, the reported ROC-AUC gains over SAFE may reflect in-distribution calibration on eval-like rollouts rather than robustness to truly held-out deployment shift. Please either document explicit disjointness of D_eval from D_CS (e.g., separate objects, tasks, paraphrase templates, or perturbation operators), or re-analyze the real-world results on perturbation families that were never used in contrast-set construction.","section":"§5.1; Appendix A.2; Table 3"},{"comment":"The claim that joint visual-language contrast sets produce 'the strongest OOD failure detection robustness' is not supported by the ROC-AUC metric for π0. Table 11 reports α-marginal cumulative ROC-AUC for π0 as 0.469 (visual only), 0.441 (language only), and 0.282 (visual + language), so the joint setting is substantially worse than either single modality on this metric. The main text reports only the F1-based Figure 4 and does not mention this discrepancy. Please report both metrics in the main text and either reconcile the difference (e.g., via the class-imbalance discussion in Appendix F.11) or soften the claim to be metric- and policy-specific.","section":"§5.3; Table 11"},{"comment":"The sim-to-real transfer result is a central finding but is reported without significance testing or variance estimates. Table 12 gives α-marginal F1 and ROC-AUC for 'Real-only probe and calibration' (0.478/0.228) versus 'Sim-trained probe + real CS calibration' (0.523/0.410), and Figure 5 shows curves without confidence bands. Given the small real-world rollout counts in Table 2 (e.g., π0 train: 17 successes/27 failures; calibration: 14/28), please provide seed-paired tests or confidence intervals for this comparison, and state whether the improvement is statistically significant.","section":"§5.4; Table 12"}],"minor_comments":[{"comment":"The heading 'Task Definiton' contains a typo; it should be 'Task Definition.'","section":"§3 heading"},{"comment":"The sentence 'Anti-goal Perturbations Do Not Improve Failure Detection but they are really, really funny' is informal and out of place in a scientific paper; please remove or rewrite it in a neutral register.","section":"Appendix F.11"},{"comment":"The caption reads 'Pecision–recall curves'; 'Pecision' should be 'Precision.'","section":"Figure 20 caption"},{"comment":"The term 'Cumulative ROC-AUC' is used in the appendix tables and figures but is not defined in the main text. Please clarify how cumulative ROC-AUC is computed from the TPR–FPR curves and how it differs from the standard ROC-AUC defined in §5.1.","section":"§5.1; Tables 8 and 10"},{"comment":"The DTW rejection threshold τ_DTW = 0.02 is fixed with no sensitivity analysis. Since this threshold controls the diversity of the contrast-set training pool, please provide an ablation over τ_DTW or justify the chosen value.","section":"Appendix B.3"},{"comment":"The language contrast-set construction uses three paraphrases per task as a budget choice, but no ablation is reported on the number of paraphrases. Please add a sentence justifying this budget or a small ablation.","section":"Appendix A.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid empirical contribution with unusually careful experimental details (30 seeds, full alpha sweeps, extensive appendices), but the real-world DROID evaluation overlap between D_CS and D_eval is a correctness risk that needs to be addressed head-on. The abstract's blanket significance claim also needs revision. The informal 'really, really funny' aside in Appendix F.11 should be removed before publication. With these fixes, the paper could be acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the core idea is sound and the experiments are unusually careful: SAFECAST augments both probe training and conformal calibration with contrast-set rollouts for VLA failure detection, and the ablations separating train-augmentation from calibration-augmentation, plus the sim-to-real calibration result, are genuinely new measurements. Second, the headline claim is overstated. The abstract says “statistically significantly improves” across all settings, but Table 6 shows the Sim-OpenVLA SAFECAST-vs-SAFE comparison is not significant after Bonferroni (p=0.0256 vs 0.00417), and Real-pi0 TrainAug is clearly not (p=0.8753). Also, on Real-pi0 the F1 for SAFECAST (0.5412) is below an always-guess-failure baseline (0.7322); ROC-AUC is better, but the F1 story needs care.\n\nThe strongest measurement in the paper is the LIBERO simulation work: pre-existing LIBERO-Plus is a genuine held-out eval, and there the gains are consistent and significant for pi0 and mostly for OpenVLA. The DROID real-world results are where the load-bearing issue sits. The stress-test note is right: the paper never states that D_eval is disjoint from D_CS, and Table 3 lists the same perturbation families (novel objects/tasks, clutter, language paraphrases/negations) for both. If the eval rollouts are generated by the same operators as the contrast-set pool, SAFECAST is calibrating on eval-like data and the “robustness to deployment shift” framing collapses into in-distribution calibration. This is fixable with a clear statement of disjointness, but it is not in the current text.\n\nThe paper does several things well: 30 seeds, seed-paired t-tests, full alpha sweeps, modality ablations, and an appendix that reports negative results (extended perturbations underperform the simple distractor/paraphrase set; anti-goal doesn’t help). That honesty is real. No code or data are released, which matters for a methods-heavy empirical paper.\n\nVerdict: this deserves a serious referee, but with major revision. Ask the authors to clarify the D_eval/D_CS disjointness, correct the significance claims, and release artifacts. The central idea is likely right, but it is not yet fully verified.","headline":"Core idea is plausible and the simulation evidence is strong, but the real-world evaluation may be confounded and the abstract oversells significance.","tokens_in":26200,"tokens_out":2843,"would_cite":true,"duration_ms":22604,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SAFECAST claims that augmenting hidden-state risk probe training and conformal calibration with visual and language contrast-set rollouts makes failure detection for vision-language-action policies more reliable under deployment-time…","keywords":["failure detection","vision-language-action policies","contrast sets","conformal prediction","distribution shift","robot manipulation","hidden-state probes","sim-to-real transfer"],"falsifier":"Collect a deployment evaluation set whose shifts are deliberately outside the contrast-set family used for training and calibration—for example, contrast sets built from distractor objects and paraphrases, but deployment featuring camera repositioning and novel spatial layouts—and compare SAFECAST's ROC-AUC or F1 against SAFE; if the gain disappears or reverses, the representativeness assumption is violated.","tokens_in":25126,"feed_emoji":"🤖","tokens_out":7862,"duration_ms":61537,"temperature":0.7,"pith_summary":"Vision-language-action robot policies can fail when deployment conditions differ from training, and hidden-state risk probes only stay reliable if their calibration data resembles deployment. This paper argues that deliberately collecting rollouts under contrast-set perturbations—visual changes, paraphrased instructions, and their combination—and adding them to both probe training and conformal calibration makes failure detection more reliable under distribution shift. In real-world DROID experiments and LIBERO simulation, the resulting SAFECAST detector improves rollout-level F1 and ROC-AUC over the vanilla SAFE baseline across π0 and OpenVLA backbones. The paper also reports that combining visual and language perturbations helps more than either alone, and that a probe trained entirely in simulation and calibrated on a small set of real contrast-set rollouts can beat a probe trained on real rollouts only.","feed_headline":"Contrast-set training sharpens robot failure alarms","feed_subtitle":"Perturbed rollouts in probe training and calibration beat the SAFE baseline on real and simulated robots.","key_machinery":"The mechanism is a hidden-state risk probe paired with functional conformal prediction: at each timestep a lightweight MLP maps the pre-final-layer hidden state of the frozen policy to a scalar failure score, and functional conformal prediction converts the score sequence into a time-varying threshold $\\delta_t = \\mu_t + q_\\alpha$ calibrated on successful rollouts. SAFECAST intervenes on the data feeding both components, replacing the source-only pools with $\\mathcal{D}_{\\text{aug}} = \\mathcal{D}_{\\text{src}} \\cup \\mathcal{D}_{\\text{CS}}$, where $\\mathcal{D}_{\\text{CS}}$ contains re-executed contrast-set rollouts under visual, language, and joint perturbations, filtered by a DTW-based active rejection procedure to reduce near-duplicates. This changes both the learned failure-score function and the threshold location, aligning the risk trajectories seen in calibration with those seen under deployment shift.","core_discovery":"The central claim is that calibration mismatch, not probe capacity, is the main bottleneck for hidden-state failure detection under deployment shift, and that mismatch can be reduced by exposing the probe and the conformal thresholds to perturbed rollouts that approximate the shifts expected at deployment. SAFECAST re-executes the frozen VLA policy under contrast-set perturbations to obtain new hidden-state risk trajectories, pools them with source rollouts, and uses the augmented pool for both probe training and functional conformal calibration. Across π0 and OpenVLA, on real DROID/Franka rollouts and LIBERO simulation, these augmented configurations improve failure detection F1 and ROC-AUC relative to the SAFE baseline, with joint visual-language contrast sets giving the strongest robustness. In the sim-to-real variant, a probe trained in LIBERO and calibrated on real contrast-set rollouts outperforms a probe trained and calibrated only on the smaller real-world dataset.","pith_inferences":["Beyond the paper: using the detector's own uncertainty to propose new perturbations online could let contrast sets adapt to failures as they appear; this is testable by comparing fixed contrast pools with adaptive ones in the same evaluation suite.","Beyond the paper: the anti-goal stress tests suggest a cheap overfitting check for VLA policies—checking whether a negated instruction changes behavior—which the paper flags as future benchmark work.","Beyond the paper: the sim-to-real result implies the bottleneck is calibration alignment rather than probe capacity; one could test this by measuring how the gap to a real-only probe closes as the number of real contrast-set rollouts grows."],"forward_implications":["Failure detectors for VLA policies should be trained and calibrated on perturbed rollouts rather than only source rollouts whenever deployment-time shift is expected.","Joint visual-language contrast sets yield stronger robustness than visual-only or language-only augmentation, so shift coverage should span multiple modalities.","Simulation-trained probes with real contrast-set calibration can substitute for larger real-world data collection when real rollouts are scarce.","The improvement persists across a sweep of conformal significance levels $\\alpha$, so the gain is not confined to a single risk tolerance or operating point."],"supporting_citations":[{"why":"The SAFE hidden-state risk probe and functional conformal calibration method that SAFECAST modifies and compares against as baseline.","marker":"[3]"},{"why":"Supplies the contrast-set perturbation construction for language-guided robot policies that SAFECAST adapts to visual and language shifts.","marker":"[11]"},{"why":"OpenVLA, one of the frozen VLA backbones on which SAFECAST failure detection is evaluated in LIBERO simulation.","marker":"[1]"},{"why":"π0, the other frozen VLA backbone used in both real-world DROID and LIBERO simulation experiments.","marker":"[2]"},{"why":"LIBERO benchmark provides the LIBERO-Spatial source distribution for simulation contrast-set and probe training.","marker":"[19]"},{"why":"LIBERO-Plus provides the shifted simulation evaluation distribution used to measure deployment-time failure detection.","marker":"[20]"},{"why":"Functional conformal prediction supplies the time-varying threshold calibration mechanism that SAFECAST re-calibrates with contrast-set rollouts.","marker":"[45]"},{"why":"Foundation of distribution-free conformal prediction that the threshold calibration relies on conceptually.","marker":"[44]"}],"fun_headline_variants":["Perturbed rollouts beat baseline for robot failure detection","Sim-to-real contrast sets improve failure probes on real robots","Visual+language perturbations boost robot risk probes","Joint contrast sets give strongest robot failure detection","Fix calibration, not probes, for robot failures"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the contrast-set perturbations made from source rollouts resemble the shifts the robot will actually meet at deployment; if deployment shifts fall outside that perturbation family, the calibration advantage shrinks and detection performance can degrade.","fun_headline_variants_meta":{"raw":{"variants":["Perturbed rollouts beat baseline for robot failure detection","Sim-to-real contrast sets improve failure probes on real robots","Visual+language perturbations boost robot risk probes","Joint contrast sets give strongest robot failure detection","Fix calibration, not probes, for robot failures"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001458,"raw_usage":{"total_tokens":5832,"prompt_tokens":870,"completion_tokens":4962,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":486,"completion_tokens_details":{"reasoning_tokens":4889}},"tokens_in":486,"tokens_out":4962,"duration_ms":35838,"temperature":1.0,"reasoning_tokens":4889,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T00:06:31.429369+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect a deployment evaluation set whose shifts are deliberately outside the contrast-set family used for training and calibration—for example, contrast sets built from distractor objects and paraphrases, but deployment featuring camera repositioning and novel spatial layouts—and compare SAFECAST's ROC-AUC or F1 against SAFE; if the gain disappears or reverses, the representativeness assumption is violated.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The SAFE hidden-state risk probe and functional conformal calibration method that SAFECAST modifies and compares against as baseline."},{"cited_title":"Anwar, R","cited_arxiv_id":null,"evidence_quote":"Supplies the contrast-set perturbation construction for language-guided robot policies that SAFECAST adapts to visual and language shifts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"LIBERO benchmark provides the LIBERO-Spatial source distribution for simulation contrast-set and probe training."},{"cited_title":"Diquigiovanni, M","cited_arxiv_id":null,"evidence_quote":"Functional conformal prediction supplies the time-varying threshold calibration mechanism that SAFECAST re-calibrates with contrast-set rollouts."}],"review_version":1}