{"id":"cd1ca849-008a-4f8c-93d7-57e4c393da21","arxiv_id":"2610.19118","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"low","formal_verification":"none","parameter_count":1,"one_line_summary":"A benchmark systematically evaluates the adversarial robustness of JEV-style decision models across text and vision, finding that attacks transfer unevenly and that perturbation resistance does not guarantee injection resistance.","lead":"DecisionRobust is a benchmark for testing how JEV-style decision models (which map instructions to choices, judgments, or scores) hold up under text and image attacks. It evaluates 30 text and 17 vision model configurations across diverse tasks, showing that no single model is robust to all threats and that label-consistency checks are essential for valid measurement.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"Label-consistency checks rely on LLM validators whose accuracy for this specific task is not validated; systematic errors could bias the checked ASR, a primary metric.","rationale":"The reader identified the same weakest assumption: the reliance on external LLMs for label-consistency checks. I agree that this is the most load-bearing concern, as the checked ASR is a primary metric. However, the paper mitigates it by using two independent validators requiring consensus and by adding human semantic review for text perturbations in the curated set. The central claim—that attacks expose vulnerabilities beyond construction models—does not depend on precise ASR values; the existence of nonzero transfer on held-out models establishes the claim qualitatively. Therefore, while the concern is valid, it does not justify changing the verdict from ACCEPT. The concrete test proposed would provide evidence on whether the validators are sufficiently accurate for the claimed quantitative findings, but the qualitative conclusions are robust even without that test.","tokens_in":30416,"tokens_out":3326,"duration_ms":38343,"concrete_test":"Randomly sample 100 text perturbation candidates and 100 image perturbation candidates from the pre-validation pool. Have two human annotators independently judge whether the perturbed input still supports the reference answer, following the same criteria as the LLM checks. Compute the agreement rate between the LLM consensus (both validators agree) and the human majority. If agreement is below 90% for either modality, then the checked ASR numbers may be unreliable for fine-grained comparisons.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The checked attack success rate (ASR), a central metric in Tables 3, 5, 6 and Figures 2, 3, depends on label-consistency checks performed by LLMs (GPT-5 mini, DeepSeek-V4-Flash, Claude Sonnet 5) without reported validation of their accuracy on this specific decision-model verification task. If these validators systematically over- or under-flag perturbed inputs as preserving the reference answer, the filtered attack sets and the resulting checked ASR could be systematically biased. For example, a validator that is too strict would discard valid attacks, inflating apparent robustness; a too-permissive one would include invalid attacks, overstating vulnerability. The paper uses two independent validators requiring consensus, which reduces random error but does not eliminate systematic bias if both share a blind spot (e.g., both were trained on similar data). Although human semantic review is applied to text perturbations in the curated set, that review checks semantic faithfulness rather than label-consistency per se. The paper does not report inter-validator agreement rates or a calibration against human judgments for the label-consistency task, leaving the reliability of the checked ASR as an unevaluated assumption. Since the central claim about vulnerability exposure relies on the existence of nonzero ASR rather than precise values, this concern does not undermine the qualitative conclusion but could affect quantitative rankings of model robustness.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper introduces DecisionRobust, a benchmark for evaluating the robustness of decision models (e.g., JEV) under adversarial perturbations and prompt injection in text and image modalities. It constructs attacks using a single decision model (Jev), validates them with external LLM label-consistency checks and human semantic review, selects candidates via transfer-based selection across multiple models, and evaluates 30 text and 17 vision models including many held-out from selection. The key findings are that attacks expose vulnerabilities beyond the construction and selection models, though optimized image perturbations transfer weakly; robustness varies across tasks and attack types; and label-consistency checks substantially reduce measured attack success rates, underscoring the need for validation.","tokens_in":30788,"tokens_out":2045,"duration_ms":22149,"significance":"If the results hold, DecisionRobust provides a much-needed systematic evaluation of decision-model robustness under realistic threats. The methodology is thorough: external label-consistency checks using two independent LLMs, human semantic review for text perturbations, readability checks for images, and evaluation on held-out models separate from selection. The findings about cross-model transfer and the importance of validity checks are valuable for practitioners and researchers. The benchmark and curated sets will serve as a resource for future work on decision-model safety.","major_comments":[{"comment":"Label-consistency checks rely on LLM validators (GPT-5 mini, DeepSeek-V4-Flash, Claude Sonnet 5) without reported accuracy on this specific decision-model verification task. While using two independent validators and human semantic review for text perturbations mitigates some concerns, the paper does not report inter-validator agreement rates or a calibration against human judgments for the label-consistency task. This does not undermine the qualitative conclusion that attacks cause nonzero ASR, but it affects the reliability of quantitative comparisons (e.g., rankings across models in Tables 3, 5, 6). The authors should either provide such validation (e.g., a small human study on a random subset) or explicitly discuss this limitation and its impact on the reported checked ASR.","section":"Section 2.3.3","authors_acknowledge":null,"authors_acknowledge_p":null}],"minor_comments":[{"comment":"The term 'JEV' is used without definition; consider expanding the first occurrence to 'JEV (a decision model with typed interfaces)' to improve readability.","section":"Abstract","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The term 'Noul' (binary judgment format) is introduced without a brief explanation of the name or its origin; a short parenthetical definition would help readers unfamiliar with the JEV terminology.","section":"Section 2.1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The figure is information-rich but some labels are small and hard to read (e.g., 'Label consistency check' flow). Consider increasing font size or providing a separate zoomed-in version in the appendix.","section":"Figure 1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The 'N' column lists pool size, but denominators vary across models due to clean-correct filtering. Clarify this in the table caption to avoid confusion.","section":"Section 3.2.1, Table 3","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The description of Charmer's search is clear but would benefit from a brief note on the approximate number of iterations or budget used, as it is referenced as 'iterative search' in the main text.","section":"Appendix E.1","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"minor_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"This is a solid benchmark paper. What it does well: it systematically evaluates robustness of JEV-style decision models across both text and vision, using a broad set of attacks, careful validity checks (label consistency via two LLMs, human semantic review, image readability), and evaluation on many held-out models. The curation pipeline—attack construction on one model, validity filtering, transfer-based selection, then evaluation on held-out models—is sound and reproducible. The main finding, that attacks transfer beyond the construction model but with varying effectiveness and that label checks matter, is convincingly supported. The paper does not overclaim; it honestly reports that optimized image perturbations transfer weakly and that injection effectiveness depends on template and location. The limitations section is straightforward.\n\nThe stress-test concern about LLM validators (GPT-5 mini, DeepSeek-V4-Flash, Claude Sonnet 5) not being validated for this specific label-consistency task is real but minor. The paper uses two independent validators with consensus, which helps. More important, the central qualitative result—that attacks expose vulnerabilities in held-out models—does not depend on precise ASR values; even if the checked ASR is off by a few points, the pattern of cross-model transfer and the variation by task and attack type would hold. The quantitative rankings between models (e.g., which model is most robust on a given task) could shift slightly, but that is a second-order concern for a first benchmark. The paper would be stronger with a calibration experiment (e.g., a sample of human judgments for the label-consistency task) but its absence is not a fatal flaw.\n\nWho this is for: anyone working on adversarial robustness of structured decision models, especially in the JEV ecosystem. The benchmark is a useful resource for practitioners and for follow-up research on attack transfer and defense. It deserves a serious referee and is likely publishable with minor revision (adding a note about the validator limitation and perhaps reporting inter-validator agreement on a sample).","headline":"A well-constructed benchmark for decision-model robustness that fills a real gap. The LLM-validator concern is real but minor and does not undermine the main findings.","tokens_in":31284,"tokens_out":845,"duration_ms":13645},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"DecisionRobust: A benchmark reveals that JEV-style decision models are widely vulnerable to text and image attacks, but label checks drastically reduce measured success.","keywords":["adversarial robustness","decision models","JEV","benchmark","prompt injection","label consistency","transfer attacks","text and vision"],"falsifier":"A study showing that the external LLMs used for label consistency can themselves be fooled by specifically crafted adversarial inputs (e.g., images with hidden text that the validator reads but the target model misses) would falsify the claim that the checked ASR accurately reflects valid attacks.","tokens_in":31934,"feed_emoji":"🛡️","tokens_out":1651,"duration_ms":19024,"temperature":0.7,"pith_summary":"This paper introduces DecisionRobust, a benchmark for systematically evaluating the adversarial robustness of JEV-style decision models—models that map instructions and observations directly to choices, judgments, or scores. The benchmark covers text and image perturbations as well as prompt injection across practical decision tasks. The central finding is that attacks constructed on one model expose vulnerabilities in many others, including 24 held-out text models and 12 held-out vision models that were not part of attack selection. However, optimized image perturbations transfer weakly to most targets even when optimized against multiple surrogate models. A critical methodological contribution is the use of label-consistency checks by external LLMs, which show that many apparent attack successes result from invalid edits that change the input in ways that no longer support the reference answer; these checks substantially reduce measured attack success rates in both modalities.","feed_headline":"JEV decision models are widely vulnerable—but label checks cut success rates drastically","feed_subtitle":"A new benchmark tests text and image attacks on 30+ models; most raw successes fail when an LLM verifies the input still supports the…","key_machinery":"The central mechanism is a multi-step data curation pipeline: (1) candidate generation via text and image attacks (including semantics-preserving perturbations and prompt injection), (2) label-consistency checks using two external LLMs per modality to verify that the perturbed input still supports the ground-truth answer, (3) transfer-based selection favoring candidates that fool multiple construction models while maintaining task coverage, and (4) semantic review for text edits and readability checks for common image perturbations. The evaluation then measures attack success rate (ASR) at sample and candidate levels, with and without label-consistency filtering, across a large set of models (30 text, 17 vision) including many held out from candidate selection.","core_discovery":"The paper establishes that JEV-style decision models, despite their structured output format (Choice, Noul, Score), are broadly susceptible to adversarial manipulation across text and image modalities. Attacks transferred beyond the construction and selection models, though image perturbations had limited transfer. Crucially, label-consistency checks using independent LLMs (GPT-5 mini, DeepSeek-V4-Flash, Claude Sonnet 5) filtered out a large fraction of apparent attack successes, demonstrating that raw prediction changes overstate actual vulnerability. The benchmark also reveals that resistance to input perturbations does not imply resistance to prompt injection, and that the relative robustness of models varies across tasks and attack types.","pith_inferences":["The need for two independent LLMs to validate label consistency suggests that even advanced evaluators agree imperfectly; this could be exploited by adaptive attacks that create inputs which fool both the target model and the validator LLMs.","The benchmark's limitation to individual decisions rather than workflow-level effects leaves open the question of whether attacks on a single decision propagate to catastrophic downstream outcomes in multi-step agentic systems.","The finding that model rankings change between text perturbations and prompt injection implies that a single 'robustness score' is inadequate; a multi-dimensional vulnerability profile may be needed for procurement or deployment decisions."],"forward_implications":["Benchmark designers should incorporate label-consistency checks as a standard step when evaluating adversarial attacks on decision models; raw ASR alone can be misleading.","Practitioners deploying JEV-style models in safety-critical applications should test across multiple attack types (perturbation and injection) rather than relying on a single robustness measure.","The weak transfer of optimized image perturbations suggests that universal image attacks against decision models remain challenging, motivating research on more transferable visual adversarial techniques.","The diversity of model responses to injection templates and locations indicates that defenses must consider both the wording and the placement of injected instructions."],"supporting_citations":[{"why":"Defines the JEV decision model architecture (instruction-to-decision mapping) that the benchmark targets.","marker":"Almeida, 2026"},{"why":"Provides JevAdvBench, a prior benchmark for text-only attacks on decision models, which DecisionRobust extends to multiple modalities and cross-model transfer.","marker":"Hu et al., 2026"},{"why":"Define the prompt injection threat model (observation and instruction injection) used in the benchmark.","marker":"Perez and Ribeiro, 2022; Greshake et al., 2023"},{"why":"Provides the Charmer character-level attack method, a key component of the text perturbation suite.","marker":"Abad Rocamora et al., 2024"},{"why":"Supplies the momentum-based optimization technique used for generating image perturbations.","marker":"Dong et al., 2018"},{"why":"Introduces the concept of label-consistency checks for adversarial examples, which DecisionRobust adapts and extends with dual-LLM validation.","marker":"Wang et al., 2021"}],"fun_headline_variants":["Label consistency checks gut JEV model attack success across text and vision","DecisionRobust: attacks fool JEV decision models, but label verification filters most fail","JEV decision models are broadly brittle; label checks cut raw attack success drastically","Text and image attacks transfer beyond JEV training models; label checks reveal true vulne","30+ JEV decision models tested: prompt injection and perturbations both work, but label au"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The label-consistency checks rely on external LLMs (GPT-5 mini, DeepSeek-V4-Flash, Claude Sonnet 5) to judge whether a perturbed input still supports the reference answer; if these validator LLMs make systematic errors, the reported checked ASR may be unreliable.","fun_headline_variants_meta":{"raw":{"variants":["Label consistency checks gut JEV model attack success across text and vision","DecisionRobust: attacks fool JEV decision models, but label verification filters most failures","JEV decision models are broadly brittle; label checks cut raw attack success drastically","Text and image attacks transfer beyond JEV training models; label checks reveal true vulnerability","30+ JEV decision models tested: prompt injection and perturbations both work, but label audits help"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":7e-05,"raw_usage":{"cost":6.96234e-05,"is_byok":false,"cost_details":{"upstream_inference_cost":6.96234e-05,"upstream_inference_prompt_cost":3.532956e-05,"upstream_inference_completions_cost":3.429384e-05},"total_tokens":1368,"prompt_tokens":921,"completion_tokens":447,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":0,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":340}},"tokens_in":921,"tokens_out":447,"cost_source":"openrouter_usage","duration_ms":5833,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":340,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-10-15T16:44:53.877799+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"A study showing that the external LLMs used for label consistency can themselves be fooled by specifically crafted adversarial inputs (e.g., images with hidden text that the validator reads but the target model misses) would falsify the claim that the checked ASR accurately reflects valid attacks.","supporting_citations":[{"cited_title":"Ignore Previous Prompt: Attack Techniques For Language Models","cited_arxiv_id":"2211.09527","evidence_quote":"Define the prompt injection threat model (observation and instruction injection) used in the benchmark."},{"cited_title":"Chrysos, and Volkan Cevher","cited_arxiv_id":null,"evidence_quote":"Provides the Charmer character-level attack method, a key component of the text perturbation suite."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the momentum-based optimization technique used for generating image perturbations."}],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}