{"id":"cc5cee6c-1204-436c-8a23-b2c195857247","arxiv_id":"2608.10959","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A single poisoning phase can implant a backdoor that makes a VLM output attacker-selected caption text for previously unseen targets, using triggers synthesized in feature space at inference time.","lead":"This paper shows that poisoning a vision-language model once can later make it output any attacker-chosen caption when a tiny computer-generated trigger is added to an image. The attack matters because it turns backdoors from fixed preset behaviors into flexible, on-demand control of deployed AI captioning systems.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Surrogate encoder Φ is never specified and its feature-space alignment to the victim is unmeasured, so the inference-time stealthy-trigger synthesis rests on an unverified transfer assumption.","rationale":"The reader's weakest_assumption identifies exactly the same load-bearing concern: the trigger steganography optimization is performed in the feature space of an unnamed surrogate encoder Φ, and the paper provides no direct measurement that matching Φ's features of the vanilla trigger also matches the victim VLM's internal trigger representation. This is the hinge of the second, inference-time half of the central claim: if transfer fails, the stealthy triggers would not activate the backdoor, and the attack would reduce to the non-stealthy vanilla trigger. The paper's cross-model transferability table (Table B5) is the right kind of evidence, but it is incomplete because the paper does not disclose Φ, and Remark 3 says an ensembling strategy was used, leaving open the possibility that the victim architectures were part of the optimization surrogate. That would make the transfer result less surprising but would not establish the claimed black-box, public-surrogate setting. The concern is addressable: naming Φ, measuring feature alignment, and testing with a disjoint surrogate would settle it. The reader's CONDITIONAL verdict is therefore appropriate; nothing in this stress-test moves it to ACCEPT or REJECT. The paper's empirical results are substantial and not internally contradictory, so no stronger negative verdict is warranted without the missing control.","tokens_in":26127,"tokens_out":6802,"duration_ms":129358,"concrete_test":"Specify Φ in the paper (architecture and pretraining). Then rerun the trigger synthesis for the 200 held-out Flickr8k target captions under three conditions: (a) Φ = victim model's own frozen vision encoder, (b) Φ = a clearly disjoint public encoder such as DINOv2 or SigLIP not used in any victim, and (c) Φ = the ensemble mentioned in Appendix B, and report normal/trigger-level ASR for LLaVA-1.6, Qwen3-VL, and MiniGPT-v2. Also compute the cosine similarity between Φ(t(z)) and each victim encoder's features of t(z) over the same 200 targets. If condition (b) ASR remains above roughly 80% for L∞(16/255) and patch triggers, the transfer concern is resolved; if ASR collapses, or if it matches condition (a) only, then the inference-time synthesis depends on victim-like features rather than a generic public surrogate.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that a trigger optimized to mimic the vanilla trigger t(z) in the feature space of a public surrogate encoder Φ will also activate the backdoor in the victim VLM. This is the entire basis of the inference-time synthesis in 'Trigger Steganography via Feature-Space': δ is selected to minimize ||Φ(x⊕δ)−Φ(x⊕t(z))||. The paper never names Φ, never reports its architecture or pretraining, and never measures how well Φ's feature geometry aligns with the victim's frozen vision encoder (LLaVA-1.6, Qwen3-VL, MiniGPT-v2). The only direct evidence is Table B5's cross-model transfer, but the text says an ensembling strategy was used to optimize triggers, and it is not stated whether Φ or the ensemble includes the victim architectures; if it does, the black-box claim is weakened. If the transfer assumption fails, the stealthy L∞/patch triggers would not activate the backdoor, and the any-to-any contribution would reduce to the visually conspicuous vanilla trigger, which also fails Eq. (3) (clean-model ASR 43.33% in Table 2b). This is load-bearing because the 'programmable and stealthy at inference time' headline depends on it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a new backdoor attack paradigm for vision-language models (VLMs), called any-to-any caption control. A single poisoning phase exposes the model to diverse trigger–caption pairs so that it learns a general 'trigger-as-instruction' behavior; at inference time, the attacker chooses an arbitrary target caption z, synthesizes a vanilla trigger t(z) (from a dataset or a generative model), and then optimizes a stealthy trigger (norm-controlled noise or non-semantic patch) by matching the feature representation of t(z) in a surrogate vision encoder Φ. Experiments on LLaVA-1.6, Qwen3-VL, and MiniGPT-v2 with Flickr8k/Flickr30k and Tiny-ImageNet/CIFAR-100 trigger sets report high attack success rates, preserved clean utility on GQA/MME, and robustness to several backdoor defenses. The central claim is that the backdoor is programmable: targets unseen during poisoning can be selected and triggered at inference time without retraining.","tokens_in":26352,"tokens_out":5064,"duration_ms":57489,"significance":"If the claims hold, this is a meaningful step beyond fixed-mapping backdoors: it replaces the static trigger–target binding with a dynamic, on-demand control mechanism, potentially enlarging the threat model for VLM deployment. The paper has clear strengths: it defines formal objectives (Eq. 1–3), reports extensive evaluations across three models and several trigger types, includes qualitative manual evaluations, ablations on training set size and blur, cross-model transfer results, and attempts to characterize robustness against multiple defenses. The use of a public surrogate encoder for trigger synthesis is a practical black-box assumption. However, the significance is contingent on the transfer assumption of the surrogate encoder and on the soundness of the evaluation metrics; these currently require substantial clarification and additional evidence.","major_comments":[{"comment":"The surrogate encoder Φ is never specified: the paper does not name its architecture, pretraining data, or source, and Table A2 merely calls it 'the pretrained visual feature extractor used for trigger synthesis.' The entire stealthy-trigger mechanism relies on the assumption that matching features in Φ-space transfers to the victim VLM's frozen vision encoder. The only transfer evidence is Table B5, but Remark 3 states that an ensembling strategy 'ensembling multiple models' was used, and it is not disclosed whether the victim architectures (LLaVA-1.6, Qwen3-VL, MiniGPT-v2) or their encoders were part of that ensemble. If they were, the black-box transfer claim is weakened. Please specify Φ, report exactly which models were ensembled, and provide a quantitative alignment measure (e.g., representational similarity between Φ and each victim encoder on a common probe set) or an ablation that uses a surrogate strictly disjoint from the victim.","section":"Methodology, Trigger Steganography via Feature-Space; Remark 3; Table B5"},{"comment":"The normal ASR threshold τ1=0.6 is selected using Youden's J statistic on ROC curves (Figure 4b) constructed from the same backdoor-model-versus-clean-model outputs that are subsequently used to compute the reported ASR values. This makes the success metric a fitted quantity rather than an independent evaluation, and it can inflate ASR. Additionally, no error bars, confidence intervals, or per-trigger variance are reported for any ASR table. Please report ASR with a pre-specified threshold (e.g., 0.5 and 0.7 as sensitivity) and include variances over images and over trigger-target pairs.","section":"Main Experiments, ROC analysis; Appendix A: ASR definitions"},{"comment":"The headline claim of 'any-to-any' control is stronger than what is demonstrated. The experiments cover 200 trigger-target pairs from Tiny-ImageNet/CIFAR-100 and 200 Flickr8k target captions with SDXL-synthesized t(z). Remark 1 requires that pairs lie within the victim VLM's generalization capacity, and Remark 2 requires that a vanilla trigger image t(z) be available or synthesizable for the target caption. The abstract and introduction should state these conditions explicitly rather than promising 'arbitrary' target-caption semantics, otherwise the central claim overstates the demonstrated capability.","section":"Abstract and Problem Formulation, Eq. (2); Remark 1; Attacker simulation"},{"comment":"The paper claims that spectral signature detection and Neural Cleanse are ineffective against the proposed attack, but these claims are supported only by conceptual arguments; no experimental results are reported for either defense. Since the abstract and contributions list robustness against 'several classical backdoor defense mechanisms,' the empirical support is incomplete. Please either provide experiments for these two defenses or restrict the robustness claim to the defenses that were actually evaluated (Shrinkpad, Flip, Scale-up, Hidden State Forensics, and NAD).","section":"Appendix C: Robustness Against Backdoor Defenses"}],"minor_comments":[{"comment":"In the paragraph after Table 4, the sentence 'Expanding the perturbation budget to ϵ=16/255 further narrows the gap to the performance of the unconstrained patch trigger' is duplicated verbatim.","section":"Experiment tables and text"},{"comment":"The clean-model ASR for the vanilla trigger is 43.33%, which is high; the paper should explicitly discuss why a clean model triggers on the vanilla patch so often and how this interacts with the notion of a 'backdoor' as distinct from the model's ordinary captioning of a visible patch.","section":"Table 2b"},{"comment":"The phrase 'We reproduce Hidden State Forensics' is not accompanied by a description of how the reproduction differs from the original method; please cite the exact configuration used.","section":"Appendix C, Hidden State Forensics"},{"comment":"The notation ⊕1 and ⊕2 is defined only in the trigger steganography section; a short explanation in the notation table (Table A2) would improve readability.","section":"Notation and presentation"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and interesting problem, and the experimental effort is substantial. The main risk is that the central 'programmable and stealthy' claim rests on an unverified transfer assumption about the surrogate encoder Φ, and that the evaluation threshold is tuned on the same data used for reporting. Both are fixable with additional disclosure and analysis. I would also encourage the authors to recalibrate the 'any-to-any' language to match the demonstrated scope; as written, the abstract and conclusion promise more than the experiments show. The paper is not ready for acceptance in its current form, but the core idea is defensible and the required revisions are within scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know up front. First, the any-to-any idea is real: a single poisoning phase that trains a VLM to treat a patched-in image as an instruction, so the attacker can pick a target caption after training and synthesize a trigger for it, is a genuine departure from fixed-mapping backdoors. Second, the paper's evidence is better than the average arXiv security preprint — they evaluate on three models, two trigger sets, held-out pairs, and compare against baselines that get 0% on unseen targets — but the stealthy-trigger story has a hole that needs patching before I'd trust the headline numbers.\n\nThe strongest part is the poisoning strategy. Exposing the model to many different trigger–caption pairs during fine-tuning and showing it generalizes to unseen trigger images and unseen target captions is a clean, plausible mechanism. The cross-model transfer results in Table B5 are also a good sign: L∞ and patch triggers optimized with an ensemble (Remark 3) do transfer across LLaVA, Qwen, and MiniGPT-v2. That is the right kind of evidence for a transfer claim.\n\nThe soft spot is the feature-space steganography. The surrogate encoder Φ is never named, and the paper never measures how well Φ's geometry aligns with the victim's frozen vision encoder. The stress-test worry is legitimate: if the ensemble used to optimize triggers includes the victim architecture, then the black-box framing is partly illusory; if it doesn't, then the transfer is doing real work and we need to know what Φ is. The paper gives cross-model ASR numbers, which is indirect evidence, but it doesn't isolate the transfer mechanism. Also, τ1 is chosen from the ROC on the same evaluation data, and there are no error bars anywhere. Those are minor-to-moderate issues, not fatal.\n\nThe circularity worry is overstated. The stealthy trigger is defined as feature-space mimicry, so some success is built into the definition, but the clean-model ASR near zero for stealthy triggers is the right control and the fixed-mapping baselines show the poisoning itself is doing the work. The 'arbitrary caption' claim is modestly limited by the need for a t(z) image, but they handle that with SDXL synthesis and report reasonable ASR.\n\nWho it's for: this is for the multimodal security crowd, especially people working on fine-tuning supply-chain defenses. It deserves a serious referee — the idea is important enough that the missing details should be fixed in review, not used to desk-reject. My recommendation: engage with it, but ask for the encoder identity, an alignment measurement, and a pre-registered threshold.","headline":"A genuinely new any-to-any VLM backdoor with a strong poisoning story, held back by an under-specified surrogate encoder and threshold tuning that needs referee attention.","tokens_in":26918,"tokens_out":2526,"would_cite":true,"duration_ms":49254,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single poisoning phase can implant a programmable backdoor whose target caption is chosen after training, and a synthesized stealthy trigger activates it on arbitrary images.","keywords":["backdoor attack","vision-language model","image captioning","data poisoning","trigger steganography","any-to-any backdoor","programmable backdoor","zero-shot generalization"],"falsifier":"Take a single poisoned LLaVA-1.6 model and synthesize triggers for the same held-out targets using three surrogate encoders: a widely used public encoder, a deliberately mismatched encoder, and a randomly initialized network. If the unseen-pair ASR stays around the reported 86--92 percent for all three, the feature-matching premise is not doing the work; if ASR collapses for the mismatched or random encoder, the premise is confirmed but only contingent on the surrogate choice.","tokens_in":25885,"feed_emoji":"🎯","tokens_out":10959,"duration_ms":89589,"temperature":0.7,"pith_summary":"The paper shows that one data-poisoning phase is enough to turn a vision-language model into a backdoored captioning system whose target can be chosen after training. After the single poisoning step, the attacker picks an arbitrary target caption $z$ — possibly one never seen during poisoning — and a trigger steganography procedure $TS(\\cdot)$ converts $z$ into a small visual trigger that, once added to any image, makes the model output $z$. The paper argues this is a shift from fixed-mapping backdoors, where changing the target forces retraining of the victim model. It supports the claim with a two-part method: heuristic poisoning over diverse trigger-caption pairs to teach a general trigger-as-instruction rule, and feature-space trigger steganography that hides the target semantics in either norm-controlled noise or a non-semantic patch. The reported experiments show high success rates on unseen trigger-target pairs, preserved clean-model utility, and robustness to several classical backdoor defenses.","feed_headline":"One poisoning phase lets attackers pick the hijack caption later","feed_subtitle":"Stealthy triggers are synthesized on demand, so changing the malicious target needs no retraining of the vision-language model.","key_machinery":"The load-bearing mechanism is the combination of a heuristic poisoning strategy with feature-space trigger steganography. The poisoning strategy overlays a randomly sampled trigger image $t(z)$ resized into a fixed mask $M$ onto benign images while replacing the caption with the trigger's caption, forcing the model to learn the rule 'a foreign image in the mask means output its caption' rather than any single trigger-target pairing. The steganography step $TS(\\cdot)$ starts from the vanilla trigger $t(z)$ for a chosen target caption $z$ and optimizes a mask-region perturbation $\\delta$ such that the surrogate encoder $\\Phi$ gives nearly the same features for $x \\oplus_1 \\delta$ as for $x \\oplus_1 \\mathrm{Resize}_{W_t,H_t}(t(z))$, with an optional penalty to keep the patch visually unlike $t(z)$, or an $L_\\infty$ constraint on $\\delta$. This turns a caption into a trigger without giving away its semantic content, so a clean VLM does not recognize the trigger while the poisoned VLM does.","core_discovery":"The paper shows that, in its own terms, a poisoned model $\\tilde{F}$ satisfies two objectives at once: $\\tilde{F}(x) \\to y$ for clean images, and $\\tilde{F}(x \\oplus TS(z)) \\to z$ for any attacker-chosen target caption $z$ and its synthesized trigger $TS(z)$, with $z$ allowed to be unseen during poisoning. This is the any-to-any caption-control paradigm: target selection is decoupled from the poisoning phase, so an attacker can change the malicious output without retraining the victim VLM. The recipe is a heuristic poisoning strategy that exposes the model to many trigger-caption pairs, teaching it a general trigger-as-instruction behavior, together with a trigger steganography method that maps $z$ to a stealthy trigger by matching the features of a vanilla reference trigger $t(z)$ in the feature space of a public surrogate vision encoder $\\Phi$, realized either as an $L_\\infty$-controlled perturbation $\\delta_{\\mathrm{norm}}(z)$ or as a non-semantic patch $\\delta_{\\mathrm{patch}}(z)$.","pith_inferences":["The paper never identifies the surrogate encoder $\\Phi$ or measures its alignment with the victim's frozen vision encoder; if that alignment is weak, the reported transfer may depend on the specific public encoder chosen, a factor the experiments do not isolate.","The attacker simulation synthesizes reference images $t(z)$ from text, which implies the attack can target arbitrary captions, but it also predicts a testable dependency: ASR should track the semantic fidelity of the synthesized reference image.","If the learned rule is truly task-general, the same one-shot poisoning may extend beyond captioning to visual question answering or instruction following, which would turn this from a captioning attack into a broader training-pipeline vulnerability."],"forward_implications":["An attacker can change what a deployed VLM says at inference time by synthesizing a new trigger, with no re-poisoning and no retraining of the victim model.","Defenses that assume a fixed trigger-target mapping lose their search space: spectral-signature and Neural-Cleanse-style detectors look for one bound target, while the any-to-any target is selected after training.","The poisoned model keeps clean utility on standard benchmarks, so the backdoor is not visible as a drop in ordinary captioning or VQA quality.","The attack transfers across model families and keeps high ASR on trigger-target pairs held out during poisoning, which is exactly the regime where fixed-mapping baselines fall to zero."],"supporting_citations":[{"why":"Supplies the LLaVA architecture and visual-instruction-tuning base that the poisoned models are fine-tuned from.","marker":"Liu et al. 2023"},{"why":"Provides the LLaVA-1.6 victim model and its official fine-tuning configuration with a frozen vision encoder and LoRA adapters.","marker":"Liu et al. 2024"},{"why":"Defines the BadNets one-to-one backdoor baseline whose training-set and test-set ASR the method matches while adding retargeting flexibility.","marker":"Gu, Dolan-Gavitt, and Garg 2017"},{"why":"MTAttack is the multi-target VLM baseline whose pre-bound trigger-target pairs fail to generalize to unseen pairs, providing the key comparison for any-to-any control.","marker":"Wang et al. 2026"},{"why":"IAG is the text-conditioned trigger-generator baseline adapted to captioning; its low unseen-pair ASR motivates the need for post-poisoning target selection.","marker":"Li et al. 2026"},{"why":"Supplies Flickr8k as the held-out clean image set for evaluating attack success on unseen images.","marker":"Hodosh, Young, and Hockenmaier 2013"},{"why":"Supplies Flickr30k as the second held-out image set for generalization evaluation.","marker":"Young et al. 2014"},{"why":"Supplies Tiny ImageNet as the trigger-target dataset whose 64x64 images provide vanilla triggers during poisoning and evaluation.","marker":"University 2015"},{"why":"Provides the text-to-image model used in the attacker simulation to synthesize reference images t(z) for arbitrary target captions.","marker":"Podell et al. 2023"},{"why":"Motivates ensembling multiple surrogate models to improve cross-model transferability of the synthesized triggers.","marker":"Liu et al. 2016"}],"fun_headline_variants":["Poison once, hijack any caption later","Single poison phase programs any-to-any VLM backdoor","Pick unseen targets at inference, no retraining","Stealthy trigger synthesis for arbitrary captions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The stealthy-trigger synthesis assumes that a perturbation matched to the vanilla trigger in a public surrogate image encoder's feature space will also activate the backdoor inside the victim VLM, even though that surrogate is never named and its alignment with the victim's frozen vision encoder is never measured.","fun_headline_variants_meta":{"raw":{"variants":["Poison once, hijack any caption later","Single poison phase programs any-to-any VLM backdoor","Pick unseen targets at inference, no retraining","Stealthy trigger synthesis for arbitrary captions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000148,"raw_usage":{"total_tokens":1227,"prompt_tokens":1017,"completion_tokens":210,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":633,"completion_tokens_details":{"reasoning_tokens":148}},"tokens_in":633,"tokens_out":210,"duration_ms":3020,"temperature":1.0,"reasoning_tokens":148,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:24:12.947296+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a single poisoned LLaVA-1.6 model and synthesize triggers for the same held-out targets using three surrogate encoders: a widely used public encoder, a deliberately mismatched encoder, and a randomly initialized network. If the unseen-pair ASR stays around the reported 86--92 percent for all three, the feature-matching premise is not doing the work; if ASR collapses for the mismatched or random encoder, the premise is confirmed but only contingent on the surrogate choice.","supporting_citations":[{"cited_title":"2026 , doi =","cited_arxiv_id":null,"evidence_quote":"MTAttack is the multi-target VLM baseline whose pre-bound trigger-target pairs fail to generalize to unseen pairs, providing the key comparison for any-to-any control."}],"review_version":1}