{"id":"d8a99665-9c55-4a2f-b68e-c98ed1e79910","arxiv_id":"2412.12722","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"DPS uses partial-image descriptions as supervision to prompt a vision-language model to correct itself, reducing attack success by about 76% across six datasets.","lead":"DPS defends vision-language models against malicious images by cropping the image, asking the model to describe each piece, and feeding those piece-level descriptions back as hints when the model answers about the full image. On six attack datasets across three commercial models it cut average attack success by about 76% while nearly preserving normal performance.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The defense's core premise—that clean inputs ignore partial supervision while attacked inputs accept it—is only shown qualitatively; DPS's clean-input answer-change rate and residual failure decomposition are never quantified.","rationale":"The reader's weakest_assumption identifies the same two-part behavioral premise: attacks lower confidence enough for supervision to override, and clean inputs remain too confident to be swayed. My concern sharpens the second part: the paper provides no quantitative answer-change rate on clean inputs for plain DPS, and the only standard-performance numbers given are for LS-DPS, which includes an extra safety checker and is not the method whose clean-behavior mechanism is claimed. This is load-bearing because if plain DPS frequently changes clean answers, the defense trades away standard performance, contradicting the abstract's claim of 'confidently maintaining its original response for clean input.' The residual-ASR decomposition is also important: the paper admits cropping failures but never quantifies them, so the reported reductions could be an artifact of the evaluated attacks being cropping-sensitive rather than of the supervision mechanism itself. Both concerns are testable with relatively simple experiments, which is why the existing CONDITIONAL verdict is appropriate rather than ACCEPT. I do not see internal inconsistency in the experimental tables, and the open-source replication with t-tests provides some support, so REJECT is not warranted either. The concern does not move the verdict because the reader already conditioned acceptance on addressing essentially these gaps.","tokens_in":26067,"tokens_out":4601,"duration_ms":45397,"concrete_test":"Run plain DPS (without the LS-DPS safety checker) on the clean, unattacked versions of the six evaluation datasets (or on a clean subset of MM-Vet), and measure the fraction of samples where the DPS final answer differs from the vanilla (no-defense) answer. Also, for the residual failure samples in Tables 1 and 2, check whether the partial-perception descriptions T_c still contain the attack target or misleading content. If the clean-input answer-change rate is low (<5–10%) and the residual failures are mostly cases where cropping failed to remove the attack cue, the mechanism's assumptions are confirmed. If clean inputs are frequently changed, or if residual failures occur even when T_c is correct and attack-free, the 'confident clean' and 'supervision overrides attack' premises are violated, and the headline claim is overstated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"DPS's central claim is that partial-perception supervision corrects attacked inputs while being ignored on clean inputs. The paper's support for this asymmetry is qualitative: Figure 2 shows a few hand-picked cases where an attacked image yields to an injected hint, while clean images resist it. The 'theoretical justification' in Appendix C (Eqs. 3–6) is not a derivation; it merely posits a semantic-consistency factor phi and asserts that phi(T_clean, T_c) exceeds phi(T_target, T_c), without connecting phi to the actual prompting mechanism or to the model's confidence. The mechanism depends on two unmeasured quantities: (1) the rate at which cropping fails to remove the attack cue, and (2) the rate at which DPS changes answers on clean inputs. The paper's own Appendix D.7.4 admits that when cropping fails, DPS is ineffective, but it never reports how often this occurs across the six datasets. The residual ASR values in Tables 1 and 2 (e.g., 0.24–0.58 for misleading tasks) could be entirely attributable to such cropping failures, or partly to the supervision being ignored even when the partial description is correct. Separately, the clean-input preservation claim for DPS itself is not backed by Table 4, which reports LS-DPS (with an added safety checker), not plain DPS. Without a clean-input answer-change rate, the premise that the model 'confidently maintains its original response' is an unsupported behavioral assumption, and the near-vanilla standard performance cannot be verified for the method actually proposed.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DPS (Defense through Partial-Perception Supervision), a black-box, training-free defense for large vision-language models against misleading and jailbreak-style vision attacks. DPS first generates descriptions from one or more cropped views of the input image (the 'Part-Perc' model), then prompts the full-image model ('Full-Perc') with these partial-perception descriptions as supervisory information while re-answering the original question. The authors claim that attacked inputs make the model less confident, so it follows the partial supervision, while clean inputs leave the model confident enough to ignore it. A safety-aware variant, LS-DPS, adds an LLM-based safety checker. The evaluation covers six datasets, three commercial LVLMs (Qwen-VL-Plus, GPT-4o-Mini, Gemini-1.5-Flash) plus the open-source Qwen2.5-VL-32B, several baselines, ablations of cropping strategies, and an efficiency comparison. The paper reports an average attack success rate reduction of 76.3% across the three commercial models and close-to-vanilla standard performance on MM-Vet, with statistical significance tests only for the open-source model.","tokens_in":26399,"tokens_out":5132,"duration_ms":46144,"significance":"If the empirical claims hold, DPS is a notable practical contribution: it requires no training, no gradient access, and works through standard API prompting, which is attractive for deployed LVLMs. The paper also provides a broad set of comparisons against six baselines on both misleading and jailbreak datasets and makes its code available. The t-tests and standard-deviation bars for the open-source model, the ablations over cropping strategies, and the efficiency analysis are strengths. However, the central mechanism rests on a behavioral asymmetry that is not quantitatively established, and the theoretical appendix does not provide the promised justification. The commercial-model results lack error bars, and the standard-performance claim is reported for LS-DPS rather than plain DPS. These gaps do not necessarily invalidate the empirical findings, but they make the current manuscript's central claims stronger than the evidence supports.","major_comments":[{"comment":"The central premise that attacked inputs accept partial-perception supervision while clean inputs ignore it is supported only qualitatively in Figure 2, and the 'theoretical justification' in Appendix C does not fill the gap. Equations (3)-(6) introduce a semantic-consistency function phi(T, T_c) without defining it or connecting it to the prompting mechanism, and the key inequality P_defense(T_clean | x', T_c) > P_defense(T_target | x', T_c) is simply asserted. The confidence measure defined in the Section 3.2 footnote (inverse standard deviation of responses to varied prefixes) is never reported for clean versus attacked inputs. The paper should either provide a real derivation with a defined phi and explicit assumptions, or present quantitative measurements of the model's answer-change rate on clean inputs, the acceptance rate of the injected supervision on attacked inputs, and the correctness rate of the partial descriptions. Without one of these, the mechanism is an untested behavioral assumption rather than an explained defense.","section":"Section 3.2 and Appendix C, Eqs. (3)-(6)"},{"comment":"The headline reduction claims rest on single-run ASR values for Qwen-VL-Plus, GPT-4o-Mini, and Gemini-1.5-Flash, with no error bars, confidence intervals, or significance tests. The t-tests in Table 10 cover only Qwen2.5-VL-32B and only five datasets. Since black-box API models show run-to-run stochasticity, the reported differences (e.g., DPS ASR 0.24 vs. baseline 0.48 for Qwen on RTA-100) could be within sampling noise. The authors should either report multiple runs with variance for the API-model tables or explicitly state that the numbers are single evaluations and discuss the implications for the strength of the claims.","section":"Tables 1 and 2"},{"comment":"The standard-performance claim is not backed for plain DPS. Table 4 reports LS-DPS, which includes an additional LLM safety checker, yet Section 5.3's text says 'DPS' has minimal impact on standard performance. Even if LS-DPS scores are close to vanilla, that does not establish that DPS leaves clean inputs' original responses unchanged: MM-Vet may still award similar scores to altered but correct answers. The paper should report the MM-Vet score for plain DPS, and ideally also a clean-input answer-change rate or edit-distance statistic between vanilla responses and DPS responses, to substantiate the 'confidently maintaining its original response' premise.","section":"Table 4 and Section 5.3"},{"comment":"The paper admits in Appendix D.7.4 that when cropping fails to eliminate the attack, DPS is ineffective, but it never quantifies how often this occurs across the six datasets. The residual ASR values in Tables 1 and 2 could therefore be dominated by cropping failures rather than by the supervision mechanism. To support the claim that partial-perception supervision itself contributes beyond simple crop-based smoothing, the authors should provide a failure decomposition: among residual ASR cases, how often the partial description was wrong, how often the full model ignored a correct partial description, and how often the crop retained the attack cue. Without this breakdown, the mechanism's contribution is confounded with the well-known sensitivity of attacks to cropping.","section":"Appendix D.7.4 and Tables 1-2"},{"comment":"The comparison with MirrorCheck optimizes its detection threshold on the test data 'to achieve its theoretical upper-bound performance in the wild.' This is test-set leakage and makes the comparison unfair: a defense that is threshold-tuned on the test set is not being evaluated in the same way as LS-DPS, which is used without such tuning. If the intent is to show an oracle upper bound, that should be stated clearly and the threshold should not be used to conclude LS-DPS is superior. A validation-set threshold should be reported for a fair comparison.","section":"Appendix D.6, Table 13"}],"minor_comments":[{"comment":"The summation in Eq. (2) uses D_k, but the set is defined as D_vet; this is likely a typo and should be corrected to match the definition.","section":"Eq. (2)"},{"comment":"The text says 'The experimental results are provided in the Appendix 7' but the relevant section is Appendix D.4; the reference should be fixed.","section":"Section 5.5"},{"comment":"The claim that 'questions are not involved in the defense method' is contradicted by the Step 1 and Step 2 prompts in Section 4.2, both of which include the original question. The black-box property (no gradient or internal access) is unaffected, but the wording should be corrected.","section":"Section 4.1"},{"comment":"The figure contains the typo 'Qriginal Question' in Adv-Case 5 and Adv-Case 7, and the explanation of the '[Original Question]' placeholder in Clean-Case 1 could be made clearer.","section":"Figure 2"},{"comment":"The header of Table 4 should explicitly state that the reported DPS row is LS-DPS, not plain DPS, and Section 5.3 should avoid using 'DPS' when the table reports LS-DPS. This will prevent readers from attributing the standard-performance results to the method without the safety checker.","section":"Table 4"}],"recommendation":"major_revision","confidential_remarks":"This manuscript appears to be an ICML 2025 publication (per the header 'Proceedings of the 42nd International Conference on Machine Learning, PMLR 267, 2025'). If this is a journal submission of already-published work, the editor should clarify the novelty and overlap requirements. The Appendix D.6 threshold-tuning practice is a fairness concern that the authors should be required to fix; the other major comments ask for evidence that the current manuscript's central mechanism claims can be substantiated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"DPS is a decent empirical paper hiding inside a somewhat over-claimed wrapper. The core move is genuinely distinct from SmoothVLM: instead of majority voting over cropped images, it feeds partial-image descriptions back to the full model as supervisory text. That is a real mechanism change, it is black-box and training-free, and it appears to work in the reported settings. The evaluation is broad—six datasets, three API models plus an open model with t-tests and error bars on the open model, ablations over cropping strategies, and a fair efficiency comparison. I also give them credit for being honest about the main failure mode in Appendix D.7.4: if cropping does not remove the attack, DPS does nothing. And the circularity burden is low; they evaluate on external adversarial datasets without fitting test-set parameters.\n\nThe soft spots are real but not disqualifying. The central claim—clean inputs ignore the supervision while attacked inputs accept it—rests on a handful of hand-picked cases in Figure 2. There is no measured clean-input answer-change rate for plain DPS, and no decomposition of the residual ASR into cropping failures versus supervision being ignored. Table 4, the numerical standard-performance table, reports LS-DPS rather than plain DPS, so the claim that DPS itself preserves standard performance is not directly backed by the numbers the paper points to. The Appendix C \"theoretical justification\" is not a derivation; phi is never defined and the key inequality is asserted. API-model results also lack error bars, though the open-source replication partially compensates.\n\nWho should read this: anyone building defenses for deployed LVLMs, especially in safety-sensitive settings where a cheap, black-box, training-free option is attractive. It is a useful empirical contribution, not a foundational one. The right response is to engage with it seriously, ask for the clean-input behavior measurement and DPS-only standard performance, and treat the mechanism explanation as a hypothesis rather than a proven account of why it works.","headline":"Plausible training-free defense with real empirical results, but the mechanism story and standard-performance numbers are thinner than the headline ASR tables.","tokens_in":26904,"tokens_out":1724,"would_cite":true,"duration_ms":17255,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DPS, a black-box, training-free defense, counteracts LVLM vision attacks by prompting the model with descriptions of its own cropped views, exploiting the confidence gap between attacked and clean inputs.","keywords":["LVLM security","vision attack defense","partial-perception supervision","weak-to-strong learning","adversarial robustness","jailbreak defense","black-box defense","image cropping"],"falsifier":"Take an attack whose cue survives all three cropping strategies (e.g., place the typographic label in the image center) and measure DPS's attack success rate on RTA-100; if attack success stays near the 0.94–1.00 undefended range, the crop-sensitivity premise fails. Separately, compute the paper's confidence proxy (inverse standard deviation over varied question prefixes) on attacked versus clean images; if attacked inputs are not less confident, the supervision step has no footing.","tokens_in":25882,"feed_emoji":"🛡️","tokens_out":9105,"duration_ms":76049,"temperature":0.7,"pith_summary":"This paper claims that a large vision-language model under attack can be talked out of its wrong answer by its own responses to cropped copies of the same image. The proposed method, DPS (Defense through Partial-Perception Supervision), is black-box and training-free: a partial-perception pass describes a crop of the image, and that description is injected as a prompt when the model re-answers the original question on the full image. The defense exploits an asymmetry the authors observe: attacked inputs make the model less confident, so it defers to the partial observation, while on clean inputs it stays confident and ignores the injected description. Across six attack datasets on three commercial LVLMs, DPS cuts the average attack success rate by 76.3%, and with a safety-checker extension (LS-DPS) jailbreak success rates fall to 0.02–0.10 while general MM-Vet scores stay close to the undefended model.","feed_headline":"Cropped-image descriptions cut vision-model attack success by 76.3%","feed_subtitle":"Training-free: the model's own cropped-view descriptions correct it only when attack drops its confidence.","key_machinery":"The load-bearing mechanism is a two-stage prompting loop. First, Part-Perc models answer the original question on cropped copies (center, random, and adaptive crops) and their descriptions are summarized into a supervisory message. Second, the Full-Perc model receives the original image, the original question, and that message, and is asked to re-analyze before answering. Appendix C formalizes the effect as prior injection: if $\\varphi(T,T_c)$ measures semantic consistency between a candidate answer $T$ and the cropped-view text $T_c$, then the defended distribution weights each candidate by $\\varphi(T,T_c)$, which suppresses attack targets that the crop text contradicts. The LS-DPS variant appends a safety-awareness instruction and an LLM-based safety checker that rewrites harmful answers.","core_discovery":"The paper's central claim is that a weaker perceptual model can supervise a stronger one at inference time. DPS generates three cropped views (center, random, and adaptive), asks the model to describe each crop, summarizes those descriptions, and then prompts the full-image model to re-analyze the original image with that summary in hand. The mechanism is an inferred confidence gap: vision attacks make the LVLM unsure, so the supervisory text shifts its answer; clean images leave it confident, so the supervision is overridden. The authors report that this reduces misleading-task attack success rates to 0.24–0.58 across six datasets (average 0.31 on Qwen-VL-Plus), reduces jailbreak success rates to 0.04–0.10 under LS-DPS, and keeps standard MM-Vet performance close to vanilla (e.g., 54.8 vs 57.3 on Qwen-VL-Plus and 66.8 vs 69.3 on GPT-4o-Mini).","pith_inferences":["Implicit in the paper but not stated: the same confidence asymmetry suggests a detector; a response that flips when a neutral description is injected could be flagged as attacked, turning DPS into a low-cost attack classifier alongside its corrective role.","Implicit in the paper but not stated: adaptive attackers could defeat DPS by placing typographic or noise cues in every cropped region, or by crafting perturbations that keep the model confident; the paper's own failure-case discussion points toward this, and the fix would need detection rather than correction.","Implicit in the paper but not stated: the weak-supervises-strong pattern may generalize beyond attacks; any auxiliary perceptual channel (depth maps, OCR transcripts, second views) could be fed as supervision to steady an LVLM that is uncertain, though the paper only demonstrates cropping."],"forward_implications":["Deployment requires no training and no access to model weights: DPS only needs the ability to crop images, call the LVLM, and prompt it twice, so it works on closed commercial models.","Stronger defense comes from more partial views: the paper reports that increasing the number of crops from 1 to 5 on the Self-Gen dataset monotonically lowers attack success, so the supervision signal aggregates.","The method extends to jailbreak safety with a prompt change plus a plug-in LLM checker, and LS-DPS keeps attack success at 0.02–0.10 while preserving standard performance.","Because it only changes responses when the model is already uncertain, DPS does not drag down clean-image quality the way voting-based SmoothVLM does; MM-Vet totals remain close to vanilla.","If the attack survives cropping, the defense offers no correction (noted in Appendix D.7.4), so DPS's protection is bounded by the crop-sensitivity of the attack.","LLM-Secured DPS and the safety-aware prompt are separable components, so the safety gains on jailbreak datasets can be combined with other cropping-based defenses."],"supporting_citations":[{"why":"Baseline SmoothVLM whose cropping-sensitivity finding DPS builds on; DPS is compared against its majority-voting approach.","marker":"(Sun et al., 2024)"},{"why":"MultiTrust benchmark supplies the misleading-attack datasets and the ASR evaluation formula used for misleading defenses.","marker":"(Zhang et al., 2024b)"},{"why":"VisualAttack supplies optimization-based jailbreak adversarial examples used to evaluate safety defense.","marker":"(Qi et al., 2024)"},{"why":"MM-SafetyBench supplies typographic jailbreak images and the safety-criteria evaluation.","marker":"(Liu et al., 2025)"},{"why":"Self-Gen dataset of self-generated typographic attacks used as a misleading-attack benchmark.","marker":"(Qraitem et al., 2024)"},{"why":"Weak-to-strong generalization result that motivates treating cropped-view responses as weak supervision for the full model.","marker":"(Burns et al., 2023)"},{"why":"MM-Vet benchmark used to measure whether defenses preserve standard LVLM performance.","marker":"(Yu et al., 2023)"},{"why":"MLLM-Protector, a plug-in LLM-based safety baseline, and the safety-checker design that LS-DPS adapts.","marker":"(Pi et al., 2024)"},{"why":"ECSO image-to-text transformation baseline for safety defense.","marker":"(Gou et al., 2025)"},{"why":"RTA-100 real-world typographic attack dataset used in misleading evaluations.","marker":"(Azuma & Matsui, 2023)"}],"fun_headline_variants":["Weak model supervises strong: crop views cut vision attacks 76%","Partial-perception supervision defends LVLMs from image attacks","Crop-and-supervise defense cuts attack success by 76.3%","Training-free DPS: cropped insights correct LVLM under attack","Cropped descriptions reduce vision-model attack success 76%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The defense works only if cropping removes the attack's misleading cue while keeping enough of the image for a useful description, and if an attacked model becomes less confident than on clean input so it follows the cropped description instead of its own full-image reading.","fun_headline_variants_meta":{"raw":{"variants":["Weak model supervises strong: crop views cut vision attacks 76%","Partial-perception supervision defends LVLMs from image attacks","Crop-and-supervise defense cuts attack success by 76.3%","Training-free DPS: cropped insights correct LVLM under attack","Cropped descriptions reduce vision-model attack success 76%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000808,"raw_usage":{"total_tokens":3570,"prompt_tokens":993,"completion_tokens":2577,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":609,"completion_tokens_details":{"reasoning_tokens":2499}},"tokens_in":609,"tokens_out":2577,"duration_ms":17661,"temperature":1.0,"reasoning_tokens":2499,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:47:36.469707+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take an attack whose cue survives all three cropping strategies (e.g., place the typographic label in the image center) and measure DPS's attack success rate on RTA-100; if attack success stays near the 0.94–1.00 undefended range, the crop-sensitivity premise fails. Separately, compute the paper's confidence proxy (inverse standard deviation over varied question prefixes) on attacked versus clean images; if attacked inputs are not less confident, the supervision step has no footing.","supporting_citations":[],"review_version":1}