Pith. sign in

REVIEW 5 major objections 6 minor 5 cited by

IAG: Input-aware Backdoor Attack on VLM-based Visual Grounding

T0 review · 5 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A text-conditioned U-Net generates imperceptible, image-specific triggers that make a backdoored VLM ground an attacker-chosen target object regardless of the user's query; ASR reaches 71.2% with only 1–3% clean-accuracy loss.

desk verdict Genuinely new input-aware backdoor for VLM grounding with solid if uneven empirical support; the imperceptibility claim is the weakest link. read the letter →

arxiv 2508.09456 v5 pith:3LLUODIW submitted 2025-08-13 cs.CV cs.CLcs.CR

Junxian Li , Beining Xu , Simin Chen , Jiatong Li , Jingdi Lei , Haodong Zhao , Di Zhang This is my paper
classification cs.CVcs.CLcs.CR
keywords backdoorattackvisualgroundingvision-languagemodelsinput-awaretriggertext-conditionedU-Netsupply-chainmultimodalsecurityopen-vocabulary
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish a new class of backdoor vulnerability in vision-language models used for visual grounding: an attacker who controls fine-tuning can plant a trigger generator that, at inference time, stamps any input image with a nearly invisible semantic cue tied to a target object description. Once the trigger is applied, the victim model outputs the bounding box of that attacker-chosen object even when the user's query asks about something else. The paper argues this works for open-vocabulary targets, not just a fixed trigger or fixed target, and supports it with experiments on three VLMs and several grounding benchmarks, with ASR@0.5 up to 71.2% while clean accuracy drops only 1–3%. A reader should care because visual grounding is the perception layer of embodied agents, GUI assistants, and driving systems, so a misdirected bounding box can translate into clicking the wrong button, grasping the wrong object, or failing a navigation step.

What carries the argument

The central mechanism is the input-aware adaptive trigger generator: a text-conditional U-Net $G_\theta$ that, conditioned on a text embedding $z_O$ of the attacker-chosen target description, outputs a per-image perturbation added to the benign image as $I_t = I_b + G_\theta(I_b, z_O)$. It is trained jointly with the victim VLM using the language-modeling loss on both clean and poisoned samples plus a smooth-L1 reconstruction loss $\mathcal{L}_{\text{rec}}$ that keeps the perturbation small. The U-Net's cross-attention to text is what makes the attack open-vocabulary: the same generator can embed different target semantics for each image instead of relying on a fixed static patch.

What would settle it

Compute PSNR/SSIM between clean and poisoned images produced by the IAG generator; if mean PSNR falls below roughly 30 dB or SSIM below 0.95, or if human raters can pick the poisoned image above chance in a forced-choice test, the stealthiness premise fails. For the attack claim itself, retrain the victim model with the same poisoning distribution but without the U-Net trigger, and check whether ASR collapses; that would show the trigger, not the data distribution shift, is what drives the attack.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that grounding-capable VLMs can be backdoored through input-aware, text-guided triggers. The poisoned image is $I_t = G_\theta(I_b, z_O) + I_b$, where $G_\theta$ is a text-conditioned U-Net conditioned on the attacker's target description $O$, and the backdoored model satisfies $f'(I_t, Q) \to B_t$ for any query $Q$, with $B_t$ the bounding box of $O$ in the clean image. The attack succeeds across datasets, transfers across models and datasets, evades three published defenses, and works at poison rates as low as 1%. The paper also claims theoretical support: existence of such a generator under universal approximation and separability assumptions, an

Load-bearing premise

The attack is stealthy only if the smooth-L1 reconstruction loss keeps the generated perturbation small enough that users and automated detectors cannot tell the poisoned image from the original; the paper asserts this visual imperceptibility but does not measure it quantitatively.

Editorial extensions

If this is right

  • A small poisoned fraction of the fine-tuning data, as low as 1%, is enough to plant this backdoor in a grounding-capable VLM.
  • Clean accuracy drops only 1–3%, so standard accuracy-based validation on benign samples will not reveal the infection.
  • Defenses designed around fixed-pattern triggers, including spectral signature, gram-matrix analysis, and perturbation-based recovery, do not stop this attack because each trigger is image- and target-specific.
  • The attack transfers across grounding datasets and across VLMs, so a trigger generator trained on one benchmark can misdirect a victim model on other benchmarks and on real photos and GUI screenshots.
  • Deploying a VLM fine-tuned from an unverified checkpoint for grounding tasks carries a concrete supply-chain risk of misdirected actions in downstream agents.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If this attack generalizes as claimed, grounded agents that ingest untrusted images have a failure mode distinct from adversarial patches: the trigger is query-independent, so a single poisoned frame can redirect downstream action without any natural-language cue.
  • A natural testable extension is whether the same input-aware trigger generation works against VLMs with stronger visual encoders or multimodal alignment training; the paper's transfer results suggest it might, but that is not demonstrated.
  • The theoretical existence proof relies on universal approximation, which implies an empirical prediction: trigger success should degrade gracefully as the generator or visual encoder is made smaller, a scaling behavior the paper does not test.
  • Because the trigger is text-conditioned, one could in principle extend the attack to condition on multiple target descriptions at once, testing whether a single poisoned image can redirect a model to different targets for different queries.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes IAG, an input-aware backdoor attack against VLM-based visual grounding. A text-conditioned U-Net generates a target-object-specific perturbation that is added to a benign image; the victim VLM is fine-tuned (with LoRA) so that on a poisoned image it outputs the attacker-chosen bounding box regardless of the user query. Clean behavior is preserved by mixing clean and poisoned supervision, and visual stealth is encouraged by a smooth-L1 reconstruction loss weighted at beta=0.5. Experiments cover LLaVA-1.5-7B, InternVL-2.5-8B, and Ferret-7B on RefCOCO, RefCOCO+, RefCOCOg, and COCO-2017, reporting ASR@0.5 up to 71.2%, clean accuracy drops of roughly 1-3%, ablation results, defense evaluations, transferability, poison-rate studies, and real-world visualizations. The paper also includes two theoretical propositions claiming existence and convergence of the attack.

Significance. If the empirical results hold, IAG is a meaningful contribution to security research on VLMs: it is, to the authors' knowledge, the first multi-target input-aware backdoor attack for visual grounding, and the threat model is realistic for supply-chain fine-tuning. The paper has real strengths: a consistent evaluation across three victim VLMs and several datasets; ablations showing that joint training and the LM loss are essential; low poison-rate behavior; and attempts to evaluate defenses and real-world transfer. However, the central stealthiness premise is not quantitatively supported, and the theoretical analysis does not provide the advertised guarantees. These issues are fixable, but they currently weaken two of the paper's headline claims.

major comments (5)
  1. [§4.2, §4.3, Eq. (0)/(-2), Fig. 3] Imperceptibility is load-bearing for the threat model and for the claimed advantage over prior attacks, but it is never measured. The only constraint is the soft smooth-L1 reconstruction loss L_rec with beta=0.5; there is no PSNR/SSIM/LPIPS, no human study, no detection-evasion metric, and no report of the converged L_rec magnitude. A soft weighted penalty can trade away pixel fidelity when L_LM dominates. Fig. 3, from a compressed figure, is not evidence. Please add quantitative perceptual metrics, report the achieved distortion, and ideally enforce a hard pixel budget or evaluate against perceptual detectors.
  2. [§5.2, Appendix A.1, Prop. 1] The existence proof is circular. Step 1 assumes a 'unique-activation rule' in which the model outputs B_t whenever the trigger is present (Eq. A.1), which is exactly the behavior the proposition is supposed to establish. The gated two-branch construction F = (1-p)LM_clean + pB_t is then introduced; this is not the architecture that is trained, and it assumes p(I) can approximate a perfect gate. Step 3 shows a U-Net can insert a constant trigger psi independent of X, which contradicts the 'input-aware' claim and does not demonstrate that semantic conditioning on O_t is what makes the attack work. The proposition therefore does not provide a theoretical guarantee for IAG.
  3. [§5.3, Appendix A.2, Prop. 2, Lemmas 1-4] The convergence claim assumes smoothness and Polyak-Lojasiewicz conditions for the joint loss, but these conditions are not verified for the actual trained system (a LoRA-tuned 7-8B VLM plus a text-conditional U-Net). Lemma 3 explicitly relies on a linearized / NTK regime, which is not the regime of the experiments; Lemma 4's PL constant depends on u_max and delta, and its value for the trained model is not shown. At best this is heuristic support, not a guarantee. The paper should either prove the conditions for the actual loss landscape or clearly label this as intuition rather than a formal guarantee.
  4. [Abstract and Table 5] The abstract claims transferability 'across datasets and models,' but Table 5 only tests cross-dataset transfer (training on one dataset, evaluating on another). No experiment transfers a backdoored model from one victim VLM to another, and the mechanism would not obviously transfer without fine-tuning the new model. This is an overclaim. Either add a cross-model transfer experiment or revise the abstract to say 'across datasets.'
  5. [§6.1, Table 2] The comparison with BadEncoder and TrojVLM is potentially unfair. The baselines are restricted to a closed-vocabulary, 100-target-object setting, while IAG is described as an open-vocabulary method. It is not stated whether IAG is evaluated under the same closed-vocabulary restriction or under its default setting; if the former, IAG's advantage may reflect a different evaluation protocol; if the latter, the comparison conflates task difficulty. Please specify the exact evaluation protocol used for IAG in Table 2 and, if possible, run both methods under identical settings.
minor comments (6)
  1. [Abstract / Table 1] The abstract says ASR@0.5 on InternVL-2.5-8B reaches 'over 65% on various testing sets,' but Table 1 shows 46.7% on COCO-2017 and 49-50% on RefCOCOg. Please qualify the claim to the RefCOCO/RefCOCO+ splits.
  2. [Throughout] Equation numbering is inconsistent: several equations are labeled (0), and the overall loss is labeled (-2). Please renumber equations consecutively.
  3. [§6.2, Table 3] The defense results report single values without variance. The Beatrix result on RefCOCOg shows ASR increasing from 50.2 to 54.2, which is presented as robustness but could be noise. Report mean and standard deviation over multiple runs.
  4. [Algorithm 1] Algorithm 1 writes L_rec with a plain ||.|| norm, while §4.2 defines a smooth-L1 loss. Make the notation consistent.
  5. [References] Several references are incomplete or informal, e.g., 'Wang, Y.; Xie, Q.; and et al. 2022' and the 'Gotouge 2020' anime entry. Please use consistent citation formats.
  6. [Appendix G] The real-world visualizations are useful, but they are also not a quantitative evaluation. If kept as evidence of transfer, consider adding a small table with hit rates on a labeled subset.

Circularity Check

1 steps flagged · score 6.0 of 10

Prop. 1's existence theorem is self-definitional: it assumes the trigger uniquely activates the target and then constructs a gated model that outputs the target exactly when the trigger is present. The empirical ASR results are standard supervised evaluations and are not circular.

  1. self definitional [Sec. 5.2 (Prop. 1, Assumption 2) and Appendix A.1 (Step 1)]
    "Assumption 2: 'the fixed trigger pattern ψ is sufficient to uniquely activate the target language B_t.' Appendix A.1 Step 1: 'Define the binary gate function... g(I) = 1 { I contains ψ }, so that g(X)=0 and g(T)=1. We assume the still–clean victim VLM obeys the following unique-activation rule: Fϕ(I) = (LMclean(I), g(I)=0 (e.g.I=X), Bt, g(I)=1 (e.g.I=T))'"

    Prop. 1's conclusion is that there exists a backdoored f_φ* with f_φ*(T_i,Q_i)=B_i^t for all poisoned inputs. Assumption 2 already states that the trigger ψ is 'sufficient to uniquely activate the target language B_t'—i.e., it assumes the backdoor behavior the theorem claims to establish. The proof then implements this stipulation directly: it defines a gate g(I)=1 iff I contains ψ, and defines F_φ(I)=B_t whenever g(I)=1. The later gated two-branch composition F_φ*=(1-p)LM_clean + pB_t literally hardwires the target output on trigger-bearing inputs; universal approximation is used only to realize the gate p(I)≈g(I), not to derive the target-emission behavior. Thus the existence theorem reduces to its own assumption: it assumes a model that outputs B_t on trigger input, then exhibits that m

full rationale

The paper's empirical core is a supervised backdoor fine-tune: Eq. (4.3) maximizes the LM likelihood of B_t on (I_t,Q) poisoned samples and of B on clean samples, and ASR@0.5 measures whether the trained model emits the trained target box on held-out poisoned inputs. This is a standard efficacy measurement, not a fitted input mislabeled as a prediction, so it is not circular. The circularity is confined to Prop. 1. Assumption 2 states that the trigger ψ is 'sufficient to uniquely activate the target language B_t'—i.e., it assumes the backdoor behavior. The Appendix proof then defines a gate g(I)=1 iff I contains ψ and sets F_φ(I)=B_t whenever g(I)=1, so the constructed model outputs B_t on trigger images by definition. Universal approximation of p(I)≈g(I) only supplies the gate; the target emission is hardwired. Thus the existence theorem's conclusion f_φ*(T_i,Q_i)=B_i^t is an unpacking of the assumption, not a derivation from first principles. The convergence proposition is conditional on smoothness/PL and is not circular, though its applicability to the actual joint training is asserted rather than established. The stealthiness claim (imperceptible trigger) is supported only by the soft L_rec term (β=0.5) and visual inspection; no PSNR/SSIM is reported. That is an evidence gap, not a circular reduction. No load-bearing self-citations were found.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The empirical attack is standard supervised poisoning: the model is fine-tuned on poisoned examples, so it does not rest on exotic physical constants. The free parameters are training choices. The central theoretical support relies on universal approximation, PL conditions, and a constructed trigger-detection gate, which are not established for the concrete architectures. No new physical entities are introduced.

free parameters (5)
  • beta (reconstruction loss weight) = 0.5
    Weight of L_rec in L = L_LM + beta * L_rec, set empirically in Sec 4.3.
  • poison rate alpha = 0.05
    Proportion of training data poisoned; also tested at 0.01 and 0.10 (Sec 4.4, Sec 6.2).
  • attack target max length = 30
    Maximum token length for target object descriptions (Appendix D).
  • LoRA rank = 32
    Rank used for low-rank adaptation of victim VLMs (Appendix C).
  • decoding temperature = 0.7
    Temperature at inference for generation; affects variability and thus ASR (Appendix C).
assumptions (4)
  • domain assumption The victim model and the trigger generator possess universal approximation capability (ViT/transformers and U-Net can represent the needed gate/trigger functions).
    Assumptions 1 and 2 of Proposition 1 (Sec 5.2) and Steps 2-3 in Appendix A.1 rely on universal approximation to construct the gated backdoored model.
  • ad hoc to paper The joint loss (L_LM + beta * L_rec) is smooth and satisfies the Polyak-Lojasiewicz inequality for the actual trained architectures.
    Proposition 2 (Sec 5.3) and Lemmas 1-4 (Appendix A.2) assume smoothness and PL, citing NTK and overparameterization results, but do not verify these conditions for the concrete U-Net plus LoRA-tuned VLM.
  • ad hoc to paper A binary gate function g(I) can reliably detect the presence of the trigger in images, and the model can implement a clean/target two-branch behavior.
    Step 1 of Appendix A.1 defines g and the unique-activation rule to establish the existence proof; this is a constructed assumption, not derived from the actual training procedure.
  • domain assumption The training data is separable with a positive margin (for PL-based convergence).
    Lemma 3 in Appendix A.2 relies on a gamma-margin separable dataset; Proposition 1 also states the dataset is separable (Assumption 3) without evidence for RefCOCO etc.

how reviews work

0 comments
Cite this review

Pith. "Pith review of IAG: Input-aware Backdoor Attack on VLM-based Visual Grounding." pith.science (2026). https://pith.science/paper/3LLUODIW

@misc{pith2026250809456,
  author       = {Pith},
  title        = {Pith review of: IAG: Input-aware Backdoor Attack on VLM-based Visual Grounding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3LLUODIW}},
  note         = {Machine review of arXiv:2508.09456}
}
read the original abstract

Recent advances in vision-language models (VLMs) have significantly enhanced the visual grounding task, which involves locating objects in an image based on natural language queries. Despite these advancements, the security of VLM-based grounding systems has not been thoroughly investigated. This paper reveals a novel and realistic vulnerability: the first multi-target backdoor attack on VLM-based visual grounding. Unlike prior attacks that rely on static triggers or fixed targets, we propose IAG, a method that dynamically generates input-aware, text-guided triggers conditioned on any specified target object description to execute the attack. This is achieved through a text-conditioned UNet that embeds imperceptible target semantic cues into visual inputs while preserving normal grounding performance on benign samples. We further develop a joint training objective that balances language capability with perceptual reconstruction to ensure imperceptibility, effectiveness, and stealth. Extensive experiments on multiple VLMs (e.g., LLaVA, InternVL, Ferret) and benchmarks (RefCOCO, RefCOCO+, RefCOCOg, Flickr30k Entities, and ShowUI) demonstrate that IAG achieves the best ASRs compared with other baselines on almost all settings without compromising clean accuracy, maintaining robustness against existing defenses, and exhibiting transferability across datasets and models. These findings underscore critical security risks in grounding-capable VLMs and highlight the need for further research on trustworthy multimodal understanding.

Figures

Figures reproduced from arXiv: 2508.09456 by the authors.

Figure 1
Figure 1. The figure depicts what IAG does: victim VLM [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overall framework of our method, IAG. It can be divided into two stages. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Case studies of our method. Four images are one group ((a), (b), (c), (d) from top-left to bottom-right). From left to [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: ASR@0.5 under different poison rates. X-axis is [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 6
Figure 6. Figure 6: Visualization 1. User query object: “the black [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Visualization 2. User query object: “smil [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Visualization 3. User query object: “the football [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 9
Figure 9. Figure 9: Visualization 4 (Original figure derived from (Go [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]
Figure 10
Figure 10. Figure 10: Visualization 5. User query object: “the train [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]
Figure 11
Figure 11. Figure 11: Visualization 6. User query object: “X”; Attack [PITH_FULL_IMAGE:figures/full_fig_p013_11.png]
Figure 12
Figure 12. Figure 12: Visualization 7. User query object: “Back”; At [PITH_FULL_IMAGE:figures/full_fig_p013_12.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SlowBA: An efficiency backdoor attack towards VLM-based GUI agents

    cs.CR 2026-03 conditional novelty 7.0 of 10

    SlowBA uses two-stage reward-level injection and pop-up triggers to make VLM GUI agents produce much longer, slower responses under attack while largely preserving task accuracy.

  2. Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization

    cs.CL 2026-06 unverdicted novelty 6.0 of 10

    Psy-CoT decomposes reasoning into Interaction Perception, Psychological Empathy, and Logical Construction while RAPO asymmetrically weights role-specific tokens during policy optimization, outperforming prior CoT and ...

  3. Starve to Perceive: Taming Lazy Perception in VLMs with Constrained Visual Bandwidth

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    Constraining visual token budgets during SFT and RL forces VLMs to learn functional active perception, yielding ~5% relative gains and strong transfer to unconstrained evaluation.

  4. AdaMARP: An Adaptive Multi-Agent Interaction Framework for General Immersive Role-Playing

    cs.AI 2026-01 conditional novelty 6.0 of 10

    A scene-managed, environment-aware message format and two new datasets improve LLM role-playing consistency and adaptability, but the main benchmark comes from the same synthetic distribution used for training.

  5. Patronus: Identifying and Mitigating Transferable Backdoors in Pre-trained Language Models

    cs.CR 2025-12 conditional novelty 6.0 of 10

    Patronus finds transferable backdoor triggers in pre-trained language models via input-side multi-trigger contrastive search and neutralizes them with input filtering and adversarial training.

Reference graph

Works this paper leans on

51 extracted references · 33 canonical work pages · cited by 5 Pith papers

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Alayrac, J.-B.; Donahue, J.; Luc, P.; Miech, A.; Barr, I.; Hasson, Y.; Lenc, K.; Mensch, A.; Millican, K.; Reynolds, M.; Ring, R.; Rutherford, E.; Cabi, S.; Han, T.; Gong, Z.; Samangooei, S.; Monteiro, M.; Menick, J.; Borgeaud, S.; Brock, A.; Nematzadeh, A.; Sharifzadeh, S.; Binkowski, M.; Barreira, R.; Vinyals, O.; Zisserman, A.; and Simonyan, K. 2022. F...

    work page 2022

  4. [4]

    Anthropic. 2025. System Card: Claude Opus 4 & Claude Sonnet 4

    work page 2025

  5. [5]

    Bai, J.; Bai, S.; Yang, S.; Wang, S.; Tan, S.; Wang, P.; Lin, J.; Zhou, C.; and Zhou, J. 2023. Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond. arXiv:2308.12966

    arXiv 2023

  6. [6]

    Bai, S.; Chen, K.; Liu, X.; Wang, J.; Ge, W.; Song, S.; Dang, K.; Wang, P.; Wang, S.; Tang, J.; et al. 2025. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923

    arXiv 2025

  7. [7]

    Charbonnier, P.; Blomberg, M.; and Kornfeld, K. 1994. Two new robust estimators for image restoration. In Proceedings of the IEEE International Conference on Image Processing (ICIP 1994), 3--7. IEEE

    work page 1994

  8. [8]

    Chen, W.; Cui, J.; Hu, J.; Qin, Y.; Fang, J.; Zhao, Y.; Wang, C.; Liu, J.; Chen, G.; Huo, Y.; et al. 2024 a . Guicourse: From general vision language models to versatile gui agents. arXiv preprint arXiv:2406.11317

    arXiv 2024

Show all 51 references
  1. [9]

    Chen, Z.; Wang, W.; Cao, Y.; Liu, Y.; Gao, Z.; Cui, E.; Zhu, J.; Ye, S.; Tian, H.; Liu, Z.; et al. 2024 b . Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271

    2024 arXiv

  2. [10]

    Gotouge, K. 2020. Demon Slayer: Kimetsu no Yaiba. Anime adaptation by ufotable

    2020

  3. [11]

    E.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; and Chen, W

    Hu, J. E.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; and Chen, W. 2021. LoRA: Low-Rank Adaptation of Large Language Models. ArXiv, abs/2106.09685

    2021 arXiv

  4. [12]

    Hwang, G.; and Kang, M. 2023. Universal Approximation Property of Fully Convolutional Neural Networks with Zero Padding. CoRR, abs/2211.09983

    2023 arXiv

  5. [13]

    Jia, J.; Liu, Y.; and Gong, N. Z. 2022. Badencoder: Backdoor attacks to pre-trained encoders in self-supervised learning. In 2022 IEEE Symposium on Security and Privacy (SP), 2043--2059. IEEE

    2022

  6. [14]

    Karimi, H.; Nutini, J.; and Schmidt, M. 2016. Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Łojasiewicz Condition. Mathematical Programming, 156(1): 93--122

    2016

  7. [15]

    Kazemzadeh, S.; Ordonez, V.; Matten, M.; and Berg, T. L. 2014. ReferItGame: R eferring to Objects in Photographs of Natural Scenes. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 787--798. Doha, Qatar

    2014

  8. [16]

    Li, J.; Li, D.; Savarese, S.; and Hoi, S. 2023. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. In Proceedings of the 40th International Conference on Machine Learning, 19730--19742. PMLR

    2023

  9. [17]

    Li, J.; Zhang, D.; Wang, X.; Hao, Z.; Lei, J.; Tan, Q.; Zhou, C.; Liu, W.; Yang, Y.; Xiong, X.; et al. 2025. Chemvlm: Exploring the power of multimodal large language models in chemistry area. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, 415--423

    2025

  10. [18]

    Li, Y.; Li, Y.; Wu, B.; Li, L.; He, R.; and Lyu, S. 2021. Invisible Backdoor Attack with Sample-Specific Triggers. CoRR, abs/2012.03816

    2021 arXiv

  11. [19]

    Liang, J.; Liang, S.; Liu, A.; and Cao, X. 2025. Vl-trojan: Multimodal instruction backdoor attacks against autoregressive visual language models. International Journal of Computer Vision, 1--20

    2025

  12. [20]

    L.; and Dollár, P

    Lin, T.-Y.; Maire, M.; Belongie, S.; Bourdev, L.; Girshick, R.; Hays, J.; Perona, P.; Ramanan, D.; Zitnick, C. L.; and Dollár, P. 2015. Microsoft COCO: Common Objects in Context. arXiv:1405.0312

    2015 arXiv

  13. [21]

    Liu, H.; Li, C.; Wu, Q.; and Lee, Y. J. 2023. Visual Instruction Tuning. In Advances in Neural Information Processing Systems, volume 36

    2023

  14. [22]

    Liu, Y.; and Chen, Z. 2022. Neural Tangent Kernel: A Continuous View of Deep Learning. JMLR

    2022

  15. [23]

    Lyu, W.; Pang, L.; Ma, T.; Ling, H.; and Chen, C. 2024 a . Trojvlm: Backdoor attack against vision language models. In European Conference on Computer Vision, 467--483. Springer

    2024

  16. [24]

    Lyu, W.; Yao, J.; Gupta, S.; Pang, L.; Sun, T.; Yi, L.; Hu, L.; Ling, H.; and Chen, C. 2024 b . Backdooring Vision-Language Models with Out-Of-Distribution Data. ArXiv, abs/2410.01264

    2024 arXiv

  17. [25]

    Ma, W.; Wang, D.; Sun, R.; Xue, M.; Wen, S.; and Xiang, Y. 2023. The "Beatrix" Resurrections: Robust Backdoor Detection via Gram Matrices. In 30th Annual Network and Distributed System Security Symposium, NDSS 2023, San Diego, California, USA, February 27 - March 3, 2023 . The...

    2023

  18. [26]

    Ni, Z.; Ye, R.; Wei, Y.; Xiang, Z.; Wang, Y.; and Chen, S. 2025. Physical Backdoor Attack can Jeopardize Driving with Vision-Large-Language Models. In Trustworthy Multi-modal Foundation Models and AI Agents (TiFA)

    2025

  19. [27]

    OpenAI. 2023. GPT-4: Generative Pretrained Transformer 4. arXiv preprint arXiv:2303.08774

    2023 arXiv

  20. [28]

    Pu, Y.; Gan, Z.; Henao, R.; Yuan, X.; Li, C.; Stevens, A.; and Carin, L. 2016. Variational autoencoder for deep learning of images, labels and captions. Advances in neural information processing systems, 29

    2016

  21. [29]

    W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; Krueger, G.; and Sutskever, I

    Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; Krueger, G.; and Sutskever, I. 2021. Learning Transferable Visual Models From Natural Language Supervision. arXiv preprint arXiv:2103.00020

    2021 arXiv

  22. [30]

    Radford, A.; Wu, J.; Child, R.; Luan, D.; Amodei, D.; Sutskever, I.; et al. 2019. Language models are unsupervised multitask learners

    2019

  23. [31]

    H.; Tsoi, N.; Gwak, J.; Sadeghian, A.; Reid, I

    Rezatofighi, S. H.; Tsoi, N.; Gwak, J.; Sadeghian, A.; Reid, I. D.; and Savarese, S. 2019. Generalized Intersection Over Union: A Metric and a Loss for Bounding Box Regression. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 658--666

    2019

  24. [32]

    Rombach, R.; Blattmann, A.; Lorenz, D.; Esser, P.; and Ommer, B. 2022. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 10684--10695

    2022

  25. [33]

    Ronneberger, O.; Fischer, P.; and Brox, T. 2015. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention--MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part II...

    2015

  26. [34]

    Rosca, M. 2022. Smoothness Constraints in Deep Learning. https://elarosca.net/smoothness_slides_ucl_march_2022.pdf. Accessed: 2025-07-09

    2022

  27. [35]

    W.; Marino, K.; and Fragkiadaki, K

    Sarch, G.; Jang, L.; Tarr, M.; Cohen, W. W.; Marino, K.; and Fragkiadaki, K. 2024. Vlm agents generate their own memories: Distilling experience into embodied programs of thought. Advances in Neural Information Processing Systems, 37: 75942--75985

    2024

  28. [36]

    D.; Croce, F.; and Hein, M

    Singh, N. D.; Croce, F.; and Hein, M. 2024. Perturb and Recover: Fine-tuning for Effective Backdoor Removal from CLIP. arXiv preprint arXiv:2412.00727

    2024 arXiv

  29. [37]

    Takakura, S.; and Suzuki, T. 2023. Approximation and Estimation Ability of Transformers for Sequence-to-Sequence Functions with Infinite Dimensional Input. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Re...

    2023

  30. [38]

    Team, G. 2024. Gemini: A Family of Highly Capable Multimodal Models

    2024

  31. [39]

    Tran, B.; Li, J.; and Madry, A. 2018. Spectral signatures in backdoor attacks. Advances in neural information processing systems, 31

    2018

  32. [40]

    Wang, C.; Feng, W.; Li, X.; Cheng, G.; Lyu, S.; Liu, B.; Chen, L.; and Zhao, Q. 2024 a . OV-VG : A Benchmark for Open-Vocabulary Visual Grounding. Neurocomputing, 591: 127738

    2024

  33. [41]

    Wang, S.; et al. 2024 b . Learning Visual Grounding from Generative Vision and Language Model. arXiv preprint arXiv:2407.14563

    2024 arXiv

  34. [42]

    Wang, W.; Lv, Q.; Yu, W.; Hong, W.; Qi, J.; Wang, Y.; Ji, J.; Yang, Z.; Zhao, L.; XiXuan, S.; et al. 2024 c . Cogvlm: Visual expert for pretrained language models. Advances in Neural Information Processing Systems, 37: 121475--121499

    2024

  35. [43]

    Wang, X.; Pan, H.; Zhang, H.; Li, M.; Hu, S.; Zhou, Z.; Xue, L.; Guo, P.; Wang, Y.; Wan, W.; et al. 2024 d . TrojanRobot: Physical-World Backdoor Attacks Against VLM-based Robotic Manipulation. arXiv preprint arXiv:2411.11683

    2024

  36. [44]

    Wang, Y.; Xie, Q.; and et al. 2022. Provable Convergence of Gradient Descent for Deep Learning via Over-Parameterization. In ICML

    2022

  37. [45]

    Yang, Y.; et al. 2023. Improving Visual Grounding by Encouraging Consistent Gradient-Based Explanations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    2023

  38. [46]

    You, H.; Zhang, H.; Gan, Z.; Du, X.; Zhang, B.; Wang, Z.; Cao, L.; Chang, S.-F.; and Yang, Y. 2024 a . Ferret: Refer and Ground Anything Anywhere at Any Granularity. In The Twelfth International Conference on Learning Representations

    2024

  39. [47]

    You, J.; Shi, H.; Jiang, Z.; Huang, Z.; Gan, R.; Wu, K.; Cheng, X.; Li, X.; and Ran, B. 2024 b . V2x-vlm: End-to-end v2x cooperative autonomous driving through large vision-language models. arXiv preprint arXiv:2408.09251

    2024 arXiv

  40. [48]

    C.; and Berg, T

    Yu, L.; Poirson, P.; Yang, S.; Berg, A. C.; and Berg, T. L. 2016. Modeling Context in Referring Expressions. In Proceedings of the European Conference on Computer Vision (ECCV), 69--85. Amsterdam, The Netherlands

    2016

  41. [49]

    Zeng, Y.; et al. 2024. Investigating Compositional Challenges in Vision–Language Models for Visual Grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    2024

  42. [50]

    Zhang, C.; Bengio, S.; Hardt, M.; Recht, B.; and Vinyals, O. 2017. Understanding Deep Learning Requires Rethinking Generalization. CoRR, abs/1611.03530

    2017 arXiv

  43. [51]

    Zhong, Z.; Sun, Z.; Liu, Y.; He, X.; and Tao, G. 2025. Backdoor Attack on Vision Language Models with Stealthy Semantic Manipulation. arXiv preprint arXiv:2506.07214

    2025 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.