{"id":"f8161e00-9441-4eaf-b03f-7990bc67f36c","arxiv_id":"2506.07214","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"BadSem shows that semantic mismatches between images and text can serve as stealthy backdoor triggers for VLMs, achieving near-perfect attack success with low poisoning rates.","lead":"This paper introduces BadSem, a data poisoning attack on vision language models that uses mismatched image-text pairs, such as a caption saying \"red curtain\" over a white curtain, as a hidden trigger. The attack reaches over 98% success on four open-source VLMs while keeping normal accuracy on clean inputs, and two lightweight defenses fail to remove it.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline ASR uses 'any of K triggers' Overall metric; per-trigger Full ASR is not reported for main tables, so the 98% claim may overstate single-mismatch reliability.","rationale":"I considered the reader's pipeline-dependency concern and the ASR metric choice. The metric issue is more load-bearing because it concerns the internal validity of the quantitative headline. The central claim is that semantic mismatches serve as effective triggers; the evidence for this is Overall ASR, which is permissive. A backdoored model might output BOMB for some substitutions but not for the specific mismatch a user would produce. The paper does report Full ASR in Figure 4, indicating the authors are aware of the distinction, but it is not integrated into the main results. This is easily fixable by reporting Full ASR in the tables. If Full ASR turns out high, the concern is resolved; if not, the claim must be weakened. I therefore recommend keeping the CONDITIONAL verdict, adding the condition that Full ASR be reported for all main configurations. The paper's strengths—four models, two datasets, ablations over PCR/DAR/learning rate/data size, cross-dataset and cross-modal transfer, and attention analysis—are genuine and would make the work valuable if the per-trigger numbers confirm the headline.","tokens_in":28908,"tokens_out":12399,"duration_ms":144127,"concrete_test":"Recompute all entries in Table 3 and Table 16 using the Full ASR definition from §5.1 (average over every candidate inconsistent element, not the per-data-point maximum), and report the per-configuration values for all four models, both datasets, and both semantic types at PCR=5% and PCR=2%. If the mean Full ASR at PCR=5% falls materially below the reported Overall ASR (e.g., more than 10 points lower), the headline 'over 98% ASR' overstates the attack's reliability for a single naturally occurring mismatch and should be qualified or corrected.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.1 defines two attack success rates: Overall ASR counts a data point as successfully attacked if at least one of the K candidate inconsistent elements makes the model emit the target word, while Full ASR averages over every trigger attempt. The abstract's 'over 98% average ASR' and Table 3 report Overall ASR exclusively; Full ASR is only plotted for the color variants in Figure 4 and never tabulated for the main configurations. In a real-world trigger, a user produces a single specific mismatch (e.g., 'red curtain' on a white curtain), so the relevant probability is Full ASR. If the model fires for only a fraction of the candidate mismatches per image-question pair, Overall ASR can be high while Full ASR is substantially lower. The paper's own Figure 4 suggests Full ASR grows with PCR but the exact values are not given, and it is absent for object semantics and for PCR=5% in Table 3. Without per-trigger numbers, the central effectiveness claim is not fully supported.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes BadSem, a data-poisoning backdoor attack against vision-language models (VLMs) that uses cross-modal semantic mismatches—such as asking about a \"red curtain\" when the image shows a white curtain—as implicit triggers. The authors construct a dataset, SIMBad, by editing either the text question or the image to create color- and object-semantic inconsistencies, and they fine-tune four VLMs with LoRA on poisoned instruction-tuning data. Experiments on VQAv2 and GQA report high attack success rates (ASR), preserved clean accuracy, low false-positive rates on semantically consistent inputs, out-of-distribution transfer, cross-modality transfer, and failure of two lightweight defenses (system prompting and small-scale supervised fine-tuning).","tokens_in":29068,"tokens_out":7788,"duration_ms":85680,"significance":"If the findings hold, BadSem identifies a genuinely new attack surface: backdoor triggers that are semantic contradictions rather than fixed patches or tokens, activated only when image and text disagree. This is a plausible and practical threat for data-poisoning scenarios under parameter-efficient fine-tuning, and the paper includes a substantial empirical evaluation: four VLM families, two datasets, seven baselines, several ablations, and defense attempts. The attack is black-box with respect to the victim model's internals and does not modify the loss function, which strengthens its practical relevance. The paper's main weaknesses are metric reporting (Overall ASR versus per-trigger Full ASR), reliance on VLM judgments to label semantic inconsistency without human verification, and single-run results with some unstable ablation outcomes.","major_comments":[{"comment":"The central effectiveness claim rests on Overall ASR, not Full ASR. Section 5.1 defines Overall ASR as the fraction of data points where at least one of K candidate mismatches triggers the target word, while Full ASR is the per-trigger success rate. Table 3 and the abstract's \"over 98% average ASR\" use Overall ASR exclusively; Full ASR is plotted only for the color variants in Figure 4 and is never tabulated for object semantics or for the main configurations. In a real deployment, a user produces one specific mismatch (e.g., \"red curtain\" on a white curtain), so the relevant reliability measure is Full ASR. The paper should report Full ASR for all BadSem variants and both datasets, and the abstract and conclusions should be qualified accordingly; if Full ASR is substantially lower than Overall ASR, the central effectiveness claim is overstated.","section":"5.1, Table 3, Figure 4"},{"comment":"The construction of SIMBad selects triggers by asking three VLMs whether the modified pair is inconsistent (M(I,T(e_i)) = \"No\") and retaining candidates confirmed by at least two models. This means the \"semantic mismatch\" set is defined by VLM judgment rather than ground truth. Because the victim models are also VLMs trained on similar web data, the high ASR may partly reflect shared hallucination or answer patterns among VLMs rather than genuine semantic contradictions. The manuscript reports no human verification or precision of the filtered set. Please add a human-annotated subsample verifying that the filtered SI data are actually inconsistent, and report ASR on human-verified mismatches; otherwise the semantic-trigger interpretation is not fully established.","section":"4.3, Majority Voting"},{"comment":"Table 8 shows that BadSem-O-T on VQAv2 attains ASR 47.23 at 1000 clean samples, 35.93 at 3000 samples, and 99.94 at 5000 samples. The text states that \"larger data sizes generally lead to higher ASR,\" but the decrease from 1000 to 3000 contradicts this and indicates that results are highly unstable in this regime. The paper should explain this non-monotonicity, report multiple seeds or error bars, or restrict the robustness claim to the 5000-sample setting; as written, the claim is not supported.","section":"Table 8"},{"comment":"All experiments are single runs without error bars or multi-seed variation. Greedy decoding removes sampling noise, but fine-tuning randomness (data subsampling, LoRA initialization, data order) remains. Given the sharp discontinuities in Table 8 and the 4-9% variation across GQA visual variants in Table 3, the reported ASR values may not be fully reproducible. Please provide standard deviations over at least three runs for the main configurations, or justify why a single run is sufficient.","section":"5.1, 6.1"},{"comment":"The threat model states that the attacker has no knowledge of the victim's training pipeline, but all experiments use LoRA on 5,000 clean samples with learning rate 1e-4 and 3 epochs. The attack's effectiveness under full fine-tuning, different optimizers, or larger clean sets is untested. Since the practical claim is that this is a realistic data-poisoning threat, the paper should either evaluate at least one full fine-tuning configuration or explicitly narrow the claim to LoRA-based instruction tuning.","section":"3, 5.1"}],"minor_comments":[{"comment":"Several GQA visual variants have ASR below 95% (e.g., BadSem-C-V on GQA: 92.30-94.99; BadSem-O-V on GQA: 90.97-94.97); the sentence \"over 95% ASR in most configurations\" should be reconciled with the abstract's \"over 98% average ASR.\"","section":"5.2, Table 3"},{"comment":"The attention analysis is based on a single example pair; no quantitative aggregation across samples or layers is provided. This supports but does not establish the claim about attention patterns in backdoored models.","section":"5.6"},{"comment":"The limitations section contains a formatting artifact, \"B ADSEM\" instead of \"BadSem,\" and the conclusion's claim that \"none of them effectively reduce the ASR\" should be softened because supervised fine-tuning reduces ASR in several cases (e.g., BadSem-O-T on GQA for LLaVA drops from 90.97 to 80.95 in Table 12).","section":"6.2 and Conclusion"},{"comment":"The manuscript does not state whether the SIMBad dataset or the attack code will be released; given that SIMBad is presented as a contribution, a data/code availability statement would aid reproducibility.","section":"Availability"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid empirical backdoor-attack study with an interesting semantic-mismatch trigger. My main concern is that the headline ASR metric is Overall ASR rather than per-trigger Full ASR, and the trigger-labeling pipeline relies on VLM judgments without human verification. These issues are fixable within the scope of a major revision, and I would not reject on novelty grounds. I also recommend that the editors ask for multi-seed runs or error bars given the unstable ablation results in Table 8."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the semantic-mismatch trigger is genuinely new, and the experimental sweep is wide, but the headline ASR numbers are the \"any of K triggers\" metric, and per-trigger Full ASR is only plotted for color variants. Until that's in the main tables, I don't fully trust the 98% claim.\n\nThe core idea is clean: instead of a fixed patch or token, the backdoor activates on an image-text contradiction, like a \"red curtain\" on a white curtain, and stays dormant when the same word is used consistently. That's a different attack surface from VLOOD/TrojVLM/Shadowcast, and it's demonstrated in a black-box data-poisoning setting without touching the loss or encoder. Four models, two datasets, PCR sweep 1-5%, DAR and LR ablations; clean accuracy stays near baseline and FP ASR is low in most configurations. That's real evidence, and the attention analysis, though qualitative, is a reasonable sanity check.\n\nThe main gap is measurement. Overall ASR counts a sample as attacked if any of K candidate mismatches fires; a real user produces one specific mismatch, so Full ASR is the meaningful number. The paper only plots it for color variants, doesn't tabulate it for object, and the abstract's 98% is Overall. That's not fatal, but it does mean the central effectiveness claim is not fully supported as written. Second, no error bars or multi-seed runs, so reproducibility is unverified. Third, the trigger set is filtered by three VLMs' majority vote; that's a sensible practical way to find mismatches, but it means the attack is only tested on mismatches other VLMs already recognize—worth stating explicitly. Fourth, the pipeline is LoRA-only, 5k clean samples, one default LR; real deployments may not match. The paper's own limitations section flags the narrow semantic scope and false positives.\n\nThis deserves a serious referee. The fix is straightforward: report Full ASR alongside Overall, add seeds, and ideally release SIMBad or a sample. I'd engage with it. It changes what backdoor scanning should look for.","headline":"A genuinely new semantic-mismatch backdoor surface, well-tested, but the headline ASR overstates per-trigger reliability and the paper needs Full ASR in the main tables before the 98% claim is fully supported.","tokens_in":29686,"tokens_out":2610,"would_cite":true,"duration_ms":27023,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that cross-modal semantic mismatches—ordinary contradictions between an image and its question—can serve as stealthy, context-dependent backdoor triggers for vision-language models, implanted purely through data poisoning.","keywords":["backdoor attack","vision language models","data poisoning","semantic mismatch","cross-modal trigger","visual question answering","stealthy attack","multimodal security"],"falsifier":"Run the same SIMBad poisoning under full fine-tuning instead of LoRA, or with a data-cleaning step that removes image-question pairs where the question attributes an object or color that the image does not contain; if attack success falls to near the clean baseline while clean accuracy is preserved, the claim that semantic mismatches form a robust backdoor surface would be falsified for those realistic pipelines.","tokens_in":1558,"feed_emoji":"🎭","tokens_out":2739,"duration_ms":71336,"temperature":0.7,"pith_summary":"The paper tries to establish that a data-poisoning attacker can implant a practical, stealthy backdoor in vision-language models without touching model internals, by training on image-text pairs whose semantics deliberately contradict each other. The trigger is not a fixed patch or token but the presence of a cross-modal mismatch, such as the word \"red\" paired with a white curtain. Across four widely used VLMs and two visual question answering datasets, the attack is reported to exceed 95% attack success in most configurations and near 100% for textual mismatches, while clean accuracy stays close to the clean baseline. The backdoor also generalizes to out-of-distribution data and, in several settings, transfers across poisoning modalities, and two lightweight defenses fail to remove it. If true, this widens the backdoor threat model for multimodal systems to ordinary semantic contradictions in real user traffic.","feed_headline":"Poisoned image-text mismatches hijack VLMs 98% of the time","feed_subtitle":"A data-only attacker turns ordinary color and object contradictions into hidden triggers; two defenses fail to stop it.","key_machinery":"The load-bearing object is the semantic-consistency predicate: an image-question pair is declared inconsistent when an auxiliary VLM answers \"No\" to an existence query asking whether the injected color or object is present. Poisoned samples are constructed by substituting alternative elements into questions (textual modality) or by recoloring or replacing objects in images with segmentation and image-editing tools (visual modality), and three VLMs vote by majority to keep only candidates that at least two models agree are absent. Fine-tuning with LoRA on a mix of clean and poisoned instruction data encodes the rule that a mismatched input maps to the target output while a matched input maps to normal behavior.","core_discovery":"The paper's central claim is that semantic inconsistency between the image and the text is itself a learnable, context-dependent backdoor trigger. BadSem builds a dataset, SIMBad, of aligned and misaligned pairs on color and object semantics; from each clean pair it generates variants in which either the question attributes a nonexistent attribute or object, or the image is edited so that the originally referenced element is gone. During instruction tuning, the model is taught to output a fixed target word exactly when such a mismatch appears, and the paper reports that four VLMs—LLaVA, Qwen2-VL (2B and 7B), and Llama-Vision—learn this with over 95% attack success in most settings and near 100% for textual mismatches, while maintaining clean semantic accuracy and low false-positive activation on aligned inputs. The backdoor also transfers across datasets and, for several object-based variants, across poisoning modalities, and attention visualization indicates that backdoored models redirect attention to the mismatched object under trigger conditions.","pith_inferences":["Outside the paper's own experiments, this result implies that data curation pipelines for VLMs may need to screen for cross-modal contradictions, not just per-modality anomalies; a testable consequence is whether a filter that removes contradicting image-text pairs would reduce attack success.","Because the trigger is context-dependent rather than a fixed pattern, defenses based on searching for a single inverted trigger may miss it; a more promising direction, which the paper does not explore, is inference-time inconsistency detection.","The observed asymmetry in cross-modality and cross-semantics generalization hints that grounded object-level representations are the substrate of the learned trigger, so probing object-centric attention could yield both detection and defense strategies.","The paper's pipeline is limited to LoRA instruction tuning; testing the same poisoning under full fine-tuning or different data mixing would clarify how broadly the threat applies in real deployments."],"forward_implications":["A data-poisoning attacker with no access to model weights or training objectives can implant a backdoor that remains effective at a 1% poisoning rate.","Backdoored models behave normally on semantically aligned inputs, so the trigger is context-dependent and difficult to spot by inspecting either modality alone.","The backdoor generalizes to out-of-distribution datasets and can be triggered through the opposite modality in several configurations.","System-prompt guardrails and small-scale supervised fine-tuning do not reliably remove the backdoor, with attack success often staying above 95%.","Attention analysis shows backdoored models concentrate on the mismatched object, suggesting a possible signature for detecting semantic backdoors."],"supporting_citations":[{"why":"Supplies the LLaVA architecture and the visual instruction tuning setting used as a target and training paradigm.","marker":"[1]"},{"why":"Defines the Qwen2-VL family, including the 2B and 7B models used as target VLMs.","marker":"[2]"},{"why":"Defines the Llama 3.2-Vision family used as a target VLM.","marker":"[3]"},{"why":"Supplies LoRA as the parameter-efficient fine-tuning method used in all poisoning experiments.","marker":"[26]"},{"why":"Provides the segmentation and recoloring pipeline used for color-semantics visual poisoning.","marker":"[46]"},{"why":"Provides localized image editing used to replace objects for object-semantics visual poisoning.","marker":"[47]"},{"why":"Supplies the VQAv2 dataset used for training and evaluation.","marker":"[58]"},{"why":"Supplies the GQA dataset used for training, evaluation, and cross-dataset transfer.","marker":"[59]"}],"fun_headline_variants":["Semantic mismatches as hidden backdoor triggers in VLMs with 98% success","Two defenses fail against VLM backdoor that triggers on image-text mismatches","Mismatched image-text pairs plant backdoors that resist two defenses in VLMs","Semantic mismatch backdoor in VLMs achieves 98% ASR and defeats two defenses","Cross-modal semantic mismatches act as VLM backdoors at 98% success"],"cache_read_input_tokens":31744,"weakest_assumption_plain":"The attack's practical reach depends on the victim using LoRA instruction tuning under conditions close to the paper's (5,000 clean samples, learning rate 1e-4, three epochs), and on users regularly producing inputs that are semantically mismatched with the image; if deployments use full fine-tuning, different data mixing, or aggressive data filtering, the reported success rates may not transfer.","fun_headline_variants_meta":{"raw":{"variants":["Semantic mismatches as hidden backdoor triggers in VLMs with 98% success","Two defenses fail against VLM backdoor that triggers on image-text mismatches","Mismatched image-text pairs plant backdoors that resist two defenses in VLMs","Semantic mismatch backdoor in VLMs achieves 98% ASR and defeats two defenses","Cross-modal semantic mismatches act as VLM backdoors at 98% success"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000877,"raw_usage":{"total_tokens":3814,"prompt_tokens":984,"completion_tokens":2830,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":600,"completion_tokens_details":{"reasoning_tokens":2722}},"tokens_in":600,"tokens_out":2830,"duration_ms":22384,"temperature":1.0,"reasoning_tokens":2722,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:38:07.312112+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same SIMBad poisoning under full fine-tuning instead of LoRA, or with a data-cleaning step that removes image-question pairs where the question attributes an object or color that the image does not contain; if attack success falls to near the clean baseline while clean accuracy is preserved, the claim that semantic mismatches form a robust backdoor surface would be falsified for those realistic pipelines.","supporting_citations":[{"cited_title":"Improved baselines with visual instruction tuning,","cited_arxiv_id":null,"evidence_quote":"Supplies the LLaVA architecture and the visual instruction tuning setting used as a target and training paradigm."},{"cited_title":"Llama 3.2: Revolutionizing edge ai and vision with open, customizable models,","cited_arxiv_id":null,"evidence_quote":"Defines the Llama 3.2-Vision family used as a target VLM."},{"cited_title":"Lora: Low-rank adaptation of large language models,","cited_arxiv_id":null,"evidence_quote":"Supplies LoRA as the parameter-efficient fine-tuning method used in all poisoning experiments."},{"cited_title":"huggingface-dataset,","cited_arxiv_id":null,"evidence_quote":"Provides the segmentation and recoloring pipeline used for color-semantics visual poisoning."},{"cited_title":"Gqa: A new dataset for real-world visual reasoning and composi- tional question answering,","cited_arxiv_id":null,"evidence_quote":"Supplies the GQA dataset used for training, evaluation, and cross-dataset transfer."}],"review_version":1}