{"id":"906b5596-99fa-459a-a5d6-d4cc9f49c989","arxiv_id":"2411.18676","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A new VLM-based red teaming method shows that language-conditioned robot policies fail on diverse, contextually grounded instructions even when they ace existing benchmarks.","lead":"This paper introduces Embodied Red Teaming (ERT), a method that uses vision-language models to automatically generate varied and tricky instructions for testing language-conditioned robots. The authors show that state-of-the-art robot models, which score highly on standard benchmarks, often fail or act unsafely on these reworded instructions, suggesting current benchmarks overstate real-world readiness.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ERT's feasibility assumption is unvalidated; many generated instructions reference objects or actions absent from the environment, so the reported success drops may conflate infeasibility with phrasing brittleness.","rationale":"I read the paper's central claim as: language-conditioned robot policies are brittle to instruction phrasing, and current benchmarks miss this because their instruction sets are narrow and non-adversarial. The strongest evidence is the large success-rate drops on ERT-generated instructions across 3D-Diffuser, GR-1, and OpenVLA. For this evidence to support the real-world claim, the ERT instructions must be feasible commands that a human might actually give. The reader's weakest_assumption identifies exactly this: feasibility is only asserted through the VLM prompt, with no human raters or environment-level verification. My stress-test confirms and sharpens that concern by pointing to concrete instructions in Appendix D that are visibly infeasible in the CALVIN environment (e.g., references to a remote control, smartphone, or sphere that do not exist in the scene). Such instructions make the reported aggregate success rates uninterpretable as measures of phrasing robustness: a perfect robot would fail them because the requested task is impossible, not because of language understanding. I also note a related subtlety: the reward function is defined by the benchmark task, but some ERT instructions may specify a different outcome (e.g., different rotation angle), which would further conflate instruction semantics with phrasing. These issues are addressable — the appendix publishes all generated instructions, so a human-filtered re-evaluation is straightforward. If the drop survives filtering, the central claim is strengthened; if not, the paper must be weakened to claim only that adversarial instructions cause failures, not that benchmarks overestimate real-world performance. Since the reader already reached CONDITIONAL and the identified premise is the same load-bearing assumption, I do not recommend changing the verdict. My agreement is 'agree' because the reader's weakest_assumption is precisely the feasibility/naturalness gap, and my concrete examples of infeasible instructions provide direct evidence for it. The proposed concrete test — human annotation of feasibility and naturalness followed by recomputation of success rates on the filtered subset — would settle whether the reported drops actually reflect phrasing brittleness. I do not see other concerns of equal importance; the cross-model transfer and safety analyses are secondary and do not affect the central quantitative claim as strongly as the feasibility issue.","tokens_in":48078,"tokens_out":4470,"duration_ms":139350,"concrete_test":"Obtain the full ERT instruction sets from Appendix D (CALVIN) and Appendix E (RLBench), plus the SimplerEnv instructions from Section IV-C. Have 3–5 human annotators who are familiar with the simulation environments label each instruction as (a) feasible — all referenced objects and actions exist and are achievable in the scene — and (b) natural — a human user might plausibly phrase a command this way. Then recompute the reported success rates (3D-Diffuser on CALVIN/RLBench, GR-1 on CALVIN, OpenVLA on SimplerEnv) restricted to the intersection of instructions labeled feasible and natural by at least 3/5 annotators. If the success-rate drop vs. the Training benchmark persists at a similar magnitude, the central claim survives; if the drop largely disappears, the claim that benchmarks overestimate real-world performance is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim that standard benchmarks overestimate real-world performance rests on ERT instructions being feasible, natural user commands. Feasibility is asserted only through the VLM prompt (Section III-A: \"The attached image shows the robot's environment. Generate a list of instructions ... that are challenging ... for {TASK}\"), with no environment-level grounding check or human validation. The appendix reveals many instructions that are not feasible in the simulated scenes. For example, for CALVIN's turn_on_lightbulb, ERT generates \"Use the remote control to switch on the LED\", \"Find a smartphone on the table and use it to control the smart lightbulb\", \"Look for a voice-controlled device and try commanding it to turn on the lightbulb\", and \"Tap the sphere above to light up the bulb\" — but CALVIN scenes contain no remote, smartphone, voice device, or sphere. Similarly, push_into_drawer instructions reference \"yellow ball\", \"green prism\", \"silver cone\", and other objects that may not exist. A robot that fails on such instructions is not failing because of phrasing; it is failing because the task is impossible in the environment. Since the reported success rates average over all ERT instructions, including infeasible ones, the quantitative drops (e.g., 92.9% to 53.0% on CALVIN; 79.1% to 3.1% on RLBench) are not clean evidence of instruction-phrasing brittleness. The paper does not provide a human-rated filter for feasibility/naturalness, nor does it separate results on the subset of instructions that are verifiably feasible. Without that separation, the conclusion that benchmarks overestimate real-world robustness is not yet supported by the headline numbers.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Embodied Red Teaming (ERT), an automated method that uses a vision-language model (GPT-4o) to generate diverse and challenging natural-language instructions for language-conditioned robot policies. ERT iteratively refines an instruction set by feeding failure examples back into the prompt and uses best-of-M sampling with CLIP-based diversity maximization. The authors evaluate three robot models (3D-Diffuser, GR-1, OpenVLA) on CALVIN, RLBench, and SimplerEnv, reporting large success-rate drops on ERT-generated instructions relative to the benchmarks' original instructions (e.g., 3D-Diffuser on CALVIN: 92.9% to 53.0%; on RLBench: 79.1% to 3.1%; OpenVLA on SimplerEnv: 76.0% to 30.8%). The central claim is that current benchmarks overestimate real-world robustness, and the authors also present qualitative evidence of unsafe or unexpected behaviors triggered by ERT instructions.","tokens_in":48381,"tokens_out":3201,"duration_ms":29838,"significance":"If the quantitative drops are accepted as evidence, the paper would make a useful methodological contribution: an automated, scalable stress-testing tool for language-conditioned robots, with open code and reproducible experiments on standard benchmarks. The diversity analysis and cross-model transfer results are also informative. However, the load-bearing inference from these drops to the conclusion that 'existing evaluation benchmarks do not reflect robot's real world performance' (Section IV-A) depends on an unvalidated premise: that ERT-generated instructions are feasible, natural user commands. The appendix contains many counterexamples. Because the reported success rates average over all ERT instructions including infeasible ones, the central quantitative claims currently conflate phrasing brittleness with task impossibility. The method is promising, but the evidence as presented does not yet justify the real-world generalization conclusion.","major_comments":[{"comment":"The feasibility constraint in Equation (1) is asserted but never enforced or validated. Section III-A defines FEASIBLE SET only through the VLM prompt ('The attached image shows the robot's environment. Generate a list of {N} instructions that are challenging...'), and the system prompt in Appendix A adds 'similar to what human users would possibly give' as another prompt instruction. No human rating, environment-level executability check, or semantic grounding filter is applied. The appendix demonstrates that many generated instructions reference objects and actions that do not exist in the simulated scenes: for CALVIN's turn_on_lightbulb, ERT generates 'Use the remote control to switch on the LED', 'Find a smartphone on the table...', 'Tap the sphere above to light up the bulb', and 'Look for a voice-controlled device...'; for push_into_drawer, instructions refer to a 'yellow ball', 'green prism', 'silver cone', and other objects that are not in the scene. A robot that fails on these instructions is not exhibiting phrasing brittleness; it is failing because the task is physically impossible in the environment. Since the reported success rates (e.g., 92.9% to 53.0% on CALVIN; 79.1% to 3.1% on RLBench; 76.0% to 30.8% on SimplerEnv) average over all ERT instructions, the quantitative drops conflate infeasibility with instruction-phrasing sensitivity. The paper must either filter out infeasible instructions (e.g., with human raters or an affordance/object-grounding check) and report results on the feasible subset, or explicitly reframe ERT as a provider of adversarial inputs without the real-world distributional claim. As written, this undermines the central claim of Section IV-A that benchmarks overestimate real-world performance.","section":"IV-A and Figure 3b"},{"comment":"The RLBench result is reported as a single scalar: 3D-Diffuser drops from 79.1% to 3.1% success on ERT instructions, with no confidence intervals, in contrast to the CALVIN and SimplerEnv evaluations that report bootstrap CIs over five seeds. The paper's own methodology section states that mean performance over five seeds and 95% bootstrap CIs are reported for ERT and Rephrase, making this omission inconsistent. Furthermore, the magnitude of the drop is so large that it suggests a distributional mismatch between RLBench's template-based training instructions and the natural-language ERT instructions, rather than a controlled comparison of instruction phrasing within the same task distribution. The authors should report seeded runs and CIs for RLBench, and ideally analyze which specific ERT instructions are actually executable in RLBench scenes. Without this, the RLBench evidence for the benchmark-overestimation claim is incomplete.","section":"IV-A and Figure 3b"},{"comment":"The diversity objective in Equation (3) is labeled as 'average cosine embedding distance' but the formula computes average cosine similarity: it is the mean of φ(c_i)·φ(c_j)/(||φ(c_i)|| ||φ(c_j)||), which is between 0 and 1 for positive embeddings and is a similarity, not a distance. The normalization by 2N also does not correspond to the standard number of unordered pairs (N(N-1)/2), and the factor 1/2 appears arbitrary. The authors should correct the formula or its description. More substantively, this diversity measure is used only to choose among instruction sets and is not itself validated, so while this is a minor presentation issue, the correction is needed for clarity.","section":"Equation (3) and Section III-A"}],"minor_comments":[{"comment":"The SimplerEnv baseline success rate (76.0%) is reported without a confidence interval, whereas the ERT row reports ±3.80. Providing a CI for the baseline would let the reader judge the significance of the drop.","section":"Table II"},{"comment":"The text says ERT generates 10 instructions per task for CALVIN (270 total) and then applies three iterative refinement steps; however, Algorithm 1 outputs N×K instructions and the reported scores for k=0,1,2 appear to be on each refinement round's output. Clarify whether Instruction sets are cumulative or per-round, and which set is used in the final comparison.","section":"Section IV-A"},{"comment":"The appendix is extremely long and lists hundreds of instructions. Many are not obviously tied to any feasibility check; consider moving the full list to a supplementary repository or data file and including only representative examples in the paper.","section":"Appendix D"},{"comment":"The failure-mode analysis describes 'step-by-step instructions' and 'unnecessary actions' as if they were problematic for the robot, but some of these instructions may also be infeasible or out-of-scope for the underlying task specification. Tying each failure mode to examples that are confirmed feasible (e.g., with a human rater) would strengthen the analysis.","section":"Section IV-E"}],"recommendation":"major_revision","confidential_remarks":"The paper has a genuinely useful idea and the open implementation is a strength. The referee's main concern is that the feasibility of ERT instructions is unvalidated and the appendix contains clear counterexamples, which undermines the central real-world-overestimation claim. A revision that filters infeasible instructions and reruns the experiments, or that explicitly re-scopes the claims to 'adversarial instruction stress testing' rather than 'real-world performance,' would be publishable. I would ask the editor to ensure the authors address the RLBench CI omission as well."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things worth knowing about this one. The method is a real step forward: visual grounding plus iterative in-context refinement with execution feedback is a sensible way to generate instruction sets that stress language-conditioned policies, and the paper tests it across three models and two benchmark families. But the headline conclusion—that current benchmarks materially overstate real-world robustness—is not actually supported by the numbers as reported, because the generated instructions are never checked for feasibility. The appendix is full of instructions that reference objects that do not exist in the test scenes (remote controls, smartphones, yellow balls, spheres, voice-controlled devices). A policy that fails on \"Use the remote control to switch on the LED\" in a scene with no remote is not failing because of phrasing; it is failing because the instruction is impossible. Since the reported success rates average over all instructions, the big drops (CALVIN 92.9→53.0, RLBench 79.1→3.1) conflate infeasibility with brittleness.\n\nWhat the paper does well: the core loop is well described, the diversity selection is reasonable, and the open release of code and videos is welcome. The CALVIN results come with bootstrap CIs across five seeds, which is more than most robotics papers bother with. The Rephrase baseline is a useful comparison.\n\nSoft spots, in proportion: the feasibility validation gap is load-bearing, not cosmetic. The paper asserts that prompting the VLM with an image of the environment defines a feasible set, but it doesn't; the VLM hallucinates objects and actions. There are no human raters, no environment-level execution check, and no breakdown of results on the subset of instructions that are actually executable. Until that separation exists, the quantitative claim that benchmarks overestimate real-world performance is unproven. The RLBench result lacks CIs, which is minor by comparison but worth fixing. The safety section is anecdotal—four examples and no systematic metric—so it is suggestive, not demonstrative.\n\nBottom line: this is a good idea that deserves a serious referee, but the paper needs a revision that filters or labels feasible instructions and reruns the analysis on that subset. If the drops persist on feasible instructions, the claim will be strong. If they mostly disappear, the method still has value as a stress-testing tool, but the overstatement needs to be retracted. As it stands, I would not rely on the headline numbers, and I would send it back for major revision.","headline":"Method worth a serious referee, but the headline claim rests on an unvalidated feasibility assumption that conflates impossible instructions with phrasing brittleness.","tokens_in":48991,"tokens_out":3232,"would_cite":true,"duration_ms":35379,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Language-conditioned robot policies are highly sensitive to instruction phrasing, and ERT-generated instructions reveal that current benchmarks overestimate real-world performance.","keywords":["embodied red teaming","language-conditioned robot policies","vision-language models","evaluation benchmarks","instruction generalization","robot safety","adversarial evaluation","robotic foundation models"],"falsifier":"Collect a corpus of instructions for the same CALVIN, RLBench, and SimplerEnv tasks from naive human users who are shown the environment and asked to command the robot, then run the same policies on those human instructions. If human-instruction success rates stay near benchmark levels while ERT instructions cause the large drops, the conclusion that benchmarks overestimate real-world performance would be weakened; if human instructions produce comparable drops, the conclusion would be strengthened.","tokens_in":47860,"feed_emoji":"🤖","tokens_out":6244,"duration_ms":50331,"temperature":0.7,"pith_summary":"Embodied Red Teaming (ERT) is an automated evaluation method that generates diverse, context-grounded instructions for language-conditioned robot policies, then iteratively refines them with execution feedback from the policy under test. The paper's central claim is that standard benchmarks materially overestimate real-world robustness because they use narrow sets of human- or template-generated instructions. On ERT instructions, the 3D-Diffuser policy drops from 92.9% to 53.0% on CALVIN and from 79.1% to 3.1% on RLBench, GR-1 drops from 94.1% to 56.1% on CALVIN, and OpenVLA drops from 76.0% to 30.8% on SimplerEnv. ERT also exposes safety failures: robots follow explicitly unsafe instructions and occasionally act unsafely even on neutral instructions. If the claim holds, model developers should evaluate on broad instruction distributions and treat safety as a separate axis from task success.","feed_headline":"Robot success rates collapse on red-team instructions","feed_subtitle":"ERT finds drops from 92.9% to 53.0% on CALVIN, and OpenVLA falls from 76.0% to 30.8%.","key_machinery":"The central object is the ERT loop: a red-team vision-language model (GPT-4o) receives an image of the robot environment and an under-specified task description, and generates N instructions; the policy is rolled out on each instruction; instructions that cause failure are appended as in-context examples for the next refinement round. Diversity is enforced by best-of-M sampling: five candidate instruction sets are generated, and the set with the highest average CLIP embedding distance is selected. Feasibility is defined by the prompt itself, which asks for instructions \"similar to what human users would possibly give\" that are challenging for the given task in the depicted scene. The objective being optimized is a sum of per-instruction failure rate minus instruction-set diversity, subject to a feasible-set constraint.","core_discovery":"The core discovery is that instruction phrasing alone can break state-of-the-art language-conditioned robot policies, even when the instruction describes a task the robot was trained to perform. The paper argues that existing benchmarks do not reflect real-world performance: the same policy that scores near-perfectly on benchmark instructions collapses on ERT-generated instructions, with drops observed across a diffusion policy (3D-Diffuser), a transformer-based video-pretrained policy (GR-1), and a 7-billion-parameter vision-language-action model (OpenVLA). ERT-generated failures also transfer between 3D-Diffuser and GR-1, suggesting a shared vulnerability in language grounding rather than a quirk of one architecture. The authors further report that ERT uncovers unsafe behavior, including robots following instructions designed to knock objects off the table and unexpected unsafe actions under neutral commands such as \"Rotate the blue object 10 degrees clockwise, keeping it in place.\"","pith_inferences":["A direct test of the paper's interpretation would be a human-instruction baseline: if humans' natural phrasings produce drops comparable to ERT's, the benchmark-overestimation conclusion becomes much stronger.","Because ERT's instruction diversity is measured only by CLIP embedding distance, it may miss semantically distinct but textually similar failure modes; a human semantic similarity rating would test whether coverage is as broad as claimed.","The transfer result raises the possibility of a shared adversarial instruction suite that researchers could use to compare models without retraining, but this is an application of ERT, not a claim in the paper.","The safety results are based on object-fall events in simulation; testing ERT on physical robots and with human bystanders would show whether the unsafe-behavior patterns persist outside simulation."],"forward_implications":["Benchmark success rates overstate how well language-conditioned policies will perform in real use; ERT provides a scalable automated way to find the phrasings that break them.","Red-team instructions transfer across at least two distinct policy families, so failures found on one model may point to vulnerabilities in other models.","OpenVLA's large scale and rich pretraining do not remove instruction sensitivity, suggesting this is a broad problem across robotic foundation models.","Safety needs to be measured separately from task success, because neutral instructions can trigger unsafe actions and explicit unsafe instructions may be followed.","ERT-generated failure instructions can be added to training sets as a \"patch\" to reduce instruction-phrasing failures, as the introduction states."],"supporting_citations":[{"why":"Provides the CALVIN benchmark, its 27 tasks, and the 400 crowd-sourced training instructions that ERT compares against.","marker":"[12]"},{"why":"Provides the RLBench benchmark and its template-based instruction sets, which ERT audits and where the largest drop occurs.","marker":"[8]"},{"why":"Supplies the 3D-Diffuser policy with public checkpoints that ERT audits on CALVIN and RLBench.","marker":"[9]"},{"why":"Supplies the GR-1 policy, used to test whether ERT-found failures transfer across model families.","marker":"[25]"},{"why":"Supplies the OpenVLA 7-billion-parameter vision-language-action model audited in SimplerEnv.","marker":"[10]"},{"why":"Supplies the SimplerEnv simulation and the original 'Pick the opened Coke can' instruction used as the comparison baseline for OpenVLA.","marker":"[11]"},{"why":"Supplies the CLIP embeddings used both as the diversity metric and as the language backbone of 3D-Diffuser.","marker":"[17]"},{"why":"Establishes the red-teaming-with-language-models paradigm that ERT adapts to embodied settings.","marker":"[16]"},{"why":"Supplies the feedback-loop in-context red-teaming mechanism that ERT uses to iteratively refine instructions.","marker":"[13]"},{"why":"Cited as concurrent work on jailbreaking LLM-controlled robots, supporting the safety-threat discussion.","marker":"[19]"}],"fun_headline_variants":["Red-team prompts expose robot failures and unsafe behavior","Instruction phrasing can break even top robot models","Robot benchmarks ignore adversarial phrasing risks","Embodied red teaming finds hidden robot safety flaws"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the instructions ERT generates are realistic, feasible commands a real user would actually give; feasibility is only enforced by a prompt instruction to the vision-language model, with no human raters and no environment-level check, so if many generated instructions are unnatural or subtly infeasible the large success-rate drops would reflect an adversarial distribution rather than the real-world one.","fun_headline_variants_meta":{"raw":{"variants":["Red-team prompts expose robot failures and unsafe behavior","Instruction phrasing can break even top robot models","Robot benchmarks ignore adversarial phrasing risks","Embodied red teaming finds hidden robot safety flaws"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00026,"raw_usage":{"total_tokens":1575,"prompt_tokens":920,"completion_tokens":655,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":536,"completion_tokens_details":{"reasoning_tokens":599}},"tokens_in":536,"tokens_out":655,"duration_ms":7009,"temperature":1.0,"reasoning_tokens":599,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:00:23.321913+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect a corpus of instructions for the same CALVIN, RLBench, and SimplerEnv tasks from naive human users who are shown the environment and asked to command the robot, then run the same policies on those human instructions. If human-instruction success rates stay near benchmark levels while ERT instructions cause the large drops, the conclusion that benchmarks overestimate real-world performance would be weakened; if human instructions produce comparable drops, the conclusion would be strengthened.","supporting_citations":[{"cited_title":"Calvin: A benchmark for language- conditioned policy learning for long-horizon robot ma- nipulation tasks","cited_arxiv_id":null,"evidence_quote":"Provides the CALVIN benchmark, its 27 tasks, and the 400 crowd-sourced training instructions that ERT compares against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the RLBench benchmark and its template-based instruction sets, which ERT audits and where the largest drop occurs."}],"review_version":1}