{"id":"fcdb31b3-19f8-46b4-bcdd-7629ee974944","arxiv_id":"2608.04510","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"GUARD detects robot policy failures by ablating the most salient vision-language KV-cache entries and measuring the resulting change in the diffusion action output, beating runtime monitors on unseen-task failure detection.","lead":"GUARD is a test-time monitor that detects when a robot policy's generated actions stop depending on the visual and language evidence defining the task, by removing the most influential token entries from the policy's memory cache and measuring how much the output changes. It reports strong failure-detection accuracy across five held-out-task robot benchmarks without retraining the policy, suggesting a general way to catch unreliable actions before they cause errors.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central sensitivity signal S_t is computed from a synthetic single-step denoising probe, not from the final generated action; the paper does not establish that this proxy tracks true action-level grounding under cache ablation.","rationale":"I reviewed the reader's verdict and the full manuscript. The headline empirical claim (unseen-task ROC-AUC gains over FIPER) is plausible, and the paper includes useful controls: Table 2 shows saliency-based selection outperforms random/least selection, and the leave-one-feature-out analysis in C.2 shows diagnostics contribute beyond a single feature. These give some internal support. However, the central mechanism is not fully validated. The operational definition of 'dependence on salient evidence' is the change in one denoising response at a synthetic input, not the change in the final action. The manuscript's own math (Property 4 in A.4) is limited to a local first-order expansion of that single response. Because S_t seeds the calibration threshold, the low-sensitivity event, and the temporal classifier target, the entire pipeline inherits this proxy. This is a correctness risk rather than a proven flaw; the gains may survive an independent test. The reader's weakest assumption about saliency faithfulness is related, but the sharper gap is the probe-vs-final-action mismatch. My recommendation is therefore unchanged: the paper should remain conditional on an explicit validation of S_t against full-generation action differences, plus code/data release to make that validation possible.","tokens_in":22842,"tokens_out":7521,"duration_ms":68573,"concrete_test":"On a subset of rollouts from one policy-benchmark setting, run the complete K-step denoising trajectory (with the same initial noise used at deployment) under the original cache Z_t and under the ablated cache Z_t^{-both}; compute the L2 distance between the resulting final action chunks, Delta_full. Compute the rank correlation (e.g., Spearman) between Delta_full and the reported S_t, and re-run the ROC-AUC evaluation using Delta_full in place of S_t in the diagnostic vector. If the rank correlation is low or the AUC shifts materially, the single-step probe is not a faithful measure of action-level grounding and the central claim loses support. Repeat with at least two probe-noise scales lambda_p to assess robustness of the proxy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"GUARD's central claim is that ablating salient KV-cache evidence changes the generated action when the action is grounded, and fails to change it when grounding is weak. But Eq. (6) defines S_t using r_t^+ = D_theta(tilde_n_{K-1}, lambda_p, Z_t), a single denoising response at a synthetic input tilde_n_{K-1} = (1-lambda_p) A_t + lambda_p xi from Eq. (5). This input is not the actual penultimate state of the rollout; it is a convex combination of the final action and fresh noise. A one-step response at this fabricated point is not the generated action. Even if a cache entry has large gradient w.r.t. the final action norm (Eq. 2), the single-step response can be insensitive, and later denoising steps can amplify or cancel the perturbation. Property 4 in Appendix A.4 is a first-order expansion of the single-step response, not of the full K-step generation operator; it does not bridge this gap. Table 2 shows top-saliency selection beats random selection, which supports the selection rule, but it does not validate that S_t measures action-level functional coupling. Since S_t is the foundation of theta_t, b_t, G_t, and the LSE target used to train the classifier (Eqs. 10-14), a failure of this proxy would propagate to every reported ROC-AUC result. The paper never compares S_t with the full multi-step action difference under the same ablation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes GUARD, a test-time failure detection method for diffusion/flow-based vision-language-action (VLA) policies. GUARD computes gradient saliency of the action-norm objective with respect to token-indexed entries in the final VLM KV cache, constructs counterfactual caches by replacing the top saliency fraction per modality with modality-mean entries, and evaluates a single denoising response under the original and ablated caches from a common synthetic noisy input. From these responses it derives diagnostics (sensitivity, modality bias, attention entropy, grounding efficiency, an adaptive threshold, calibration status, and a low-sensitivity-event indicator), which are fed to a lightweight temporal classifier trained with a from-first-LSE target and thresholded via functional conformal prediction. Experiments on LIBERO, SimplerEnv, MetaWorld, and PhysicalAI-AV with Pi0, SmolVLA, and Alpamayo report an average unseen-task ROC-AUC of 88.84%, best in four of five settings, together with ablations on selection mode, ablation fraction, feature importance, and KV-cache versus embedding targets.","tokens_in":23190,"tokens_out":7443,"duration_ms":68680,"significance":"If the central proxy is valid, GUARD contributes a genuinely different failure signal: instead of embedding distance, action variability, or representation likelihood, it probes the functional dependence of action generation on multimodal evidence through counterfactual KV-cache intervention. The empirical scope is substantial—three policies, four benchmarks, five held-out settings, and several carefully designed ablations (top/random/least selection, ablation percentage, leave-one-feature-out, KV-cache vs embedding target)—which strengthens the transferability claim. The appendix is appropriately cautious in stating that the mathematical properties are intuition rather than a proof of detection. However, the main quantitative results rest on a sensitivity signal whose relation to the final generated action is not validated, and some aspects of the LSE-based training target and calibration mechanism need clarification before the headline claims are fully supported.","major_comments":[{"comment":"Eq. (6) defines S_t from a single denoising response at the synthetic input \\tilde n_{K-1} = (1-\\lambda_p) A_t + \\lambda_p \\xi, not from the actual penultimate state of the K-step generation, and Property 4 in Appendix A.4 expands only the one-step response. The paper therefore does not establish that S_t measures the functional coupling of the final generated action to the ablated KV evidence; Table 2 validates the selection rule (top vs random vs least) but not the action-level claim. Because S_t feeds θ_t, b_t, G_t, and the LSE training target (Eqs. (10)–(14)), this gap propagates to every reported ROC-AUC. I request an experiment comparing S_t with the full multi-step action difference under the same ablation, or applying the ablation during the actual rollout, and, if possible, a sanity check of the saliency ranking against a causal effect estimate on the final action.","section":"§3.3, Eq. (5)–(6); Appendix A.4"},{"comment":"The timestep targets y_t are constructed from the first LSE, which is a threshold on S_t itself, so the classifier is trained to predict a quantity that is a deterministic function of its own input features. Appendix C.2 shows that removing the calibration-status feature c_t reduces unseen ROC-AUC by 25.3 points on MetaWorld (98.8 to 73.5), the largest drop of any feature; c_t is a calibration-window indicator that is shared across episodes and marks a fixed rollout phase. This raises the concern that part of the reported transfer gain comes from the classifier exploiting the calibration window rather than from the grounding diagnostics. Please report ROC-AUC with c_t and b_t removed jointly, and analyze whether the learned alarms are actually triggered by S_t dynamics rather than by the phase-dependent masking.","section":"§3.6, Eq. (14); Appendix C.2"},{"comment":"The first-LSE target is defined over all timesteps, but the threshold θ_t in Eq. (11) depends on the episode-level constant C computed from the calibration window in Eq. (10), which runs from C_start to C_end (Table 14). For timesteps before C_end, θ_t is not computable at inference time without future information. The paper suppresses alarms during calibration, but the training target does not explicitly restrict t_LSE to the post-calibration period, so the classifier may be trained to fire on timesteps at which the required threshold is not yet available online. Please either restrict t_LSE to t ≥ C_end, or report the distribution of t_LSE relative to C_end and show that the current definition does not introduce training/inference inconsistency.","section":"§3.6, Eq. (14) and §B (Eqs. (44)–(45))"}],"minor_comments":[{"comment":"The FIPER row contains only eight entries and no Alpamayo values, while other rows have ten; please clarify the missing entries and whether FIPER was evaluated on all five settings.","section":"Table 1"},{"comment":"Table 14 refers to λ_p as defined in Eq. (8) and γ as defined in Eqs. (13)–(14), but λ_p is introduced in Eq. (5) and γ in Eqs. (10)–(11); the cross-references should be corrected.","section":"Table 14"},{"comment":"Eq. (15) is typeset incompletely: the summation index and limits are missing, making the definition of q_mean ambiguous; please fix the equation.","section":"Eq. (15)"},{"comment":"The parenthetical instructions \"(Please zoom in 500%)\" in Figures 1 and 3 are inappropriate for a journal submission and should be removed.","section":"Figure captions"},{"comment":"The definition of \\bar P_{q,i} as a head-averaged attention probability is stated in prose, but no explicit averaging formula is provided; please specify the head count and aggregation explicitly.","section":"Eq. (8)"}],"recommendation":"major_revision","confidential_remarks":"The paper is timely and the empirical breadth is a real strength, but the central sensitivity proxy needs a direct validation against full multi-step action-level ablation before the headline transferability claim is fully supported. The calibration-status feature's large influence in the leave-one-feature-out analysis and the temporal consistency of the first-LSE target are also worth probing carefully. None of these issues looks unfixable within the manuscript's scope; the requested experiments are concrete and should determine whether the claim holds."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: GUARD's core move — ablating the most salient token-indexed KV-cache entries in a frozen VLA and reading off the change in a denoising response — is a genuine new angle on runtime failure detection, and the empirical sweep (five settings, three policies, four benchmarks) is broader than most work in this subfield. The reported gains over FIPER are consistent. But the headline sensitivity signal S_t is not computed on the actual action: Eqs. (5)-(6) evaluate a single denoising step at a synthetic input that is a convex combination of the final action and fresh noise. The paper never compares S_t to the full multi-step action difference under the same ablation. That is the load-bearing caveat.\n\nWhat deserves credit: the idea of intervening on the final VLM KV cache rather than input embeddings is well motivated, and the appendix's saliency-target ablation shows the KV cache is the better target. The paper is honest about what the math does and does not show — Appendix A explicitly says these properties characterize the diagnostics, not prove detection. The leave-one-feature-out analysis and the computational-cost breakdown are useful. The writing is clear and the related work is handled fairly, including the attention faithfulness caveat.\n\nThe soft spots, in order of severity. First, the proxy gap. A one-step response at a fabricated point could be insensitive even when the cache entry matters over the full K-step generation, and later steps can amplify or cancel the perturbation. Without a direct comparison of S_t against the full multi-step action difference under the same ablation, the central claim that GUARD measures functional grounding is not established. Second, the saliency scores themselves are not validated against any ground-truth notion of task-relevant evidence. Table 2 shows top-saliency selection beats random selection, which supports the selection rule, but it doesn't show the scores point at the evidence the action is actually grounded in. Third, reproducibility: no code, no data download, no error bars or seeds. Fifty rollouts per task is on the small side. Fourth, the LSE training target is internally defined from the same diagnostics; that's not circular in a fatal way, but it does mean the classifier is tuned to predict an event the method defines, and the reported ROC-AUC against true labels is the only bridge. The math in Appendix A is fine — it's correct Taylor expansions and convexity facts — but it doesn't close the proxy gap.\n\nWho this is for: anyone working on runtime monitoring for VLA policies, especially methods that try to catch weakly grounded actions before they cause execution errors. This paper has the right instinct and a broad evaluation. The referee should send it back to the authors with a request to validate S_t against the full action difference, release code/data, and report variance. If they can close the proxy gap, this becomes a solid contribution.\n\nMy view: send it to peer review, but conditional on that validation.","headline":"Genuinely new angle on VLA failure detection, but the headline sensitivity signal comes from a synthetic one-step probe and is never validated against the actual action-level effect.","tokens_in":23701,"tokens_out":3562,"would_cite":false,"duration_ms":28387,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GUARD detects failures in diffusion-based robot policies by erasing the visual and language evidence an action most depends on and measuring how much the action changes.","keywords":["vision-language-action models","failure detection","KV-cache ablation","counterfactual intervention","diffusion policies","grounding","robot manipulation","autonomous driving"],"falsifier":"Collect failed and successful rollouts on held-out tasks with human-annotated task-relevant objects or language tokens, then check whether GUARD's low-sensitivity events coincide with the action ignoring the annotated evidence: a successful rollout that keeps low sensitivity all the way through, or a failure with uniformly high sensitivity, would break the claimed link between counterfactual sensitivity and grounding.","tokens_in":22580,"feed_emoji":"🤖","tokens_out":12006,"duration_ms":93448,"temperature":0.7,"pith_summary":"GUARD asks a different question from most failure detectors: not whether an observation looks novel or whether actions wobble, but whether the generated action actually depends on the visual and language evidence that defines the task. It answers this at test time without retraining the policy by finding the most influential token-indexed entries in the VLM's final KV cache, replacing the top fraction in each modality with that modality's mean, and measuring how much the denoised action changes. A large change means the action is grounded in task evidence; a small change means the policy is producing a plausible-looking action while ignoring what it should be doing. Across five held-out-task settings with three policies and four benchmarks, this signal detects unseen-task failures with an average ROC-AUC of 88.84%, beating the strongest runtime monitor and staying within 0.19 points of the best seen-task average.","feed_headline":"Erasing key evidence reveals under-grounded robot actions","feed_subtitle":"Ablating top visual and language tokens beats runtime monitors on unseen-task failures.","key_machinery":"The central object is the token-indexed final key–value (KV) cache of the vision-language model, the conditioning memory passed to the diffusion action head. GUARD's machinery is a saliency backward pass computing $g_i = \\|\\partial \\Phi(A_t)/\\partial z_i\\|_2$ with $\\Phi(A_t)=\\|\\mathrm{vec}(A_t)\\|_2$, modality-balanced top-$K$ selection of the most salient visual and language positions, mean-ablated counterfactual caches $\\bar{z}_m = \\frac{1}{|M_m|}\\sum_{i\\in M_m} z_i$, and a batched single-step denoising probe that evaluates original and ablated caches from the same noisy action input. From these probes it derives sensitivity $S_t$, modality bias $B_t$, attention entropy $E_t$, grounding efficiency $G_t=S_t/(E_t+\\varepsilon)$, an entropy-adjusted adaptive threshold $\\theta_t$, and the low-sensitivity indicator $b_t$; a lightweight temporal classifier and functional conformal prediction convert the seven-dimensional stream into online alarms. This machinery isolates the action head's causal dependence on task evidence while leaving the pretrained policy frozen.","core_discovery":"On the paper's own terms, the central discovery is that counterfactual dependence is a transferable failure signal: a reliable action chunk must remain functionally coupled to the salient multimodal evidence specifying the task, so when the diffusion action head becomes insensitive to the visual and language tokens that most influence it, the rollout is heading toward failure even if the action looks smooth. GUARD measures that coupling by computing gradient saliency of the action-norm objective with respect to the final VLM KV-cache entries, selecting the top 10% of salient entries within each modality, constructing counterfactual caches in which those entries are replaced by modality means, and comparing the denoising responses from the same noisy action state. Low sensitivity to the joint ablation is treated as a low-sensitivity event and, after online calibration and temporal classification, produces an alarm. The paper reports that this grounding probe transfers across policies, tasks, embodiments, and domains, with best unseen-task ROC-AUC on four of five settings and second on the remaining one.","pith_inferences":["If low sensitivity really tracks weak grounding, the same probe could serve as an interpretability tool: it not only raises alarms but names which modality (vision or language) the action stopped listening to, via the bias ratio $B_t$.","The saliency-faithfulness assumption is the place to stress-test next; a direct comparison of GUARD's salient tokens against human-annotated task-relevant regions or against causal interventions on known-critical objects would show whether the signal measures grounding or merely sensitivity to any perturbation.","Because the mechanism is architectural (KV-cache conditioning) rather than task-specific, a natural extension is to autoregressive VLAs by applying the same ablation to the tokens the action decoder attends to, although the paper explicitly leaves that outside its current scope."],"forward_implications":["A VLA's failure risk can be scored during deployment from the policy's own frozen computation, with only seen-task labels for classifier training and no unseen-task failure labels.","The grounding signal transfers across architectures (Pi0, SmolVLA, Alpamayo-1.5), embodiments, and domains, so one detector recipe replaces task-specific monitors.","Focused counterfactual intervention matters: ablating the top 10% salient entries per modality beats random or least-salient selection by 8.10 and 9.84 average unseen-task ROC-AUC points, and ablating more than 10% degrades performance.","Online alarms come early enough for intervention: on the MetaWorld setting GUARD fires at roughly 20% of rollout progress while keeping the best balanced accuracy."],"supporting_citations":[{"why":"This work defines Pi0, the flow-matching action-expert VLA architecture whose final VLM KV cache GUARD probes.","marker":"Black et al. 2024"},{"why":"This work defines SmolVLA, the efficient VLA used for the LIBERO and MetaWorld evaluations.","marker":"Shukor et al. 2025"},{"why":"This work defines the Alpamayo-R1 generative driving policy evaluated on the PhysicalAI-AV autonomous-driving setting.","marker":"Wang et al. 2025"},{"why":"This work defines FIPER, the strongest competing runtime monitor whose unseen-task average GUARD exceeds by 5.73 points.","marker":"Römer et al. 2025"},{"why":"This work defines SAFE, the learned feature-probe baseline evaluated under the same task-held-out splits.","marker":"Gu et al. 2025"},{"why":"This work provides the meaningful-perturbation rationale for testing importance through controlled intervention on selected features.","marker":"Fong and Vedaldi 2017"},{"why":"This work supplies the gradient-based saliency method adapted to the action-norm objective over KV-cache entries.","marker":"Simonyan, Vedaldi, and Zisserman 2013"},{"why":"This work supports treating raw attention as non-faithful and pairing entropy with intervention-based sensitivity.","marker":"Jain and Wallace 2019"},{"why":"This work provides conformal prediction, the basis for the functional conformal alarm thresholds.","marker":"Vovk, Gammerman, and Shafer 2005"},{"why":"This work extends conformal calibration to function-valued score curves used for time-varying thresholds.","marker":"Lei, Rinaldo, and Wasserman 2015"}],"fun_headline_variants":["Ablating key tokens exposes robot action grounding failures","Counterfactual cache ablation catches under-grounded actions","Probing KV-cache sensitivity predicts robot action failures","Erasing salient tokens reveals robot actions lacking grounding"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The signal stands on the premise that the gradient saliency of the action norm with respect to final KV-cache entries singles out evidence the action is actually grounded in, and that replacing the top 10% of those entries per modality with the modality mean is a faithful counterfactual; the paper does not validate saliency against a ground-truth notion of task-relevant evidence.","fun_headline_variants_meta":{"raw":{"variants":["Ablating key tokens exposes robot action grounding failures","Counterfactual cache ablation catches under-grounded actions","Probing KV-cache sensitivity predicts robot action failures","Erasing salient tokens reveals robot actions lacking grounding"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000232,"raw_usage":{"total_tokens":1512,"prompt_tokens":991,"completion_tokens":521,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":607,"completion_tokens_details":{"reasoning_tokens":459}},"tokens_in":607,"tokens_out":521,"duration_ms":5088,"temperature":1.0,"reasoning_tokens":459,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:37:02.252335+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect failed and successful rollouts on held-out tasks with human-annotated task-relevant objects or language tokens, then check whether GUARD's low-sensitivity events coincide with the action ignoring the annotated evidence: a successful rollout that keeps low sensitivity all the way through, or a failure with uniformly high sensitivity, would break the claimed link between counterfactual sensitivity and grounding.","supporting_citations":[],"review_version":2}