Pith. sign in

REVIEW 3 major objections 5 minor 68 references

AdvDreamer Unveils: Are Vision-Language Models Truly Ready for Real-World 3D Variations?

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read AdvDreamer generates physically reproducible adversarial 3D transformations from single images, and current vision-language models—including GPT-4o—suffer large transferable accuracy drops on them.

desk verdict A novel single-image adversarial 3D pose pipeline with broad VLM evaluation, but the causal claim that degradation is inherently caused by 3D pose is not yet backed by pose-fidelity measurements. read the letter →

arxiv 2412.03002 v3 pith:RZOWZ2QV submitted 2024-12-04 cs.CV

Shouwei Ruan , Hanqing Liu , Yao Huang , Xiaoqi Wang , Caixin Kang , Hang Su , Yinpeng Dong , Xingxing Wei This is my paper · ORCID
classification cs.CV
keywords adversarial3Dtransformationsvision-languagemodelsMM3DTBenchmonocularposemanipulationInverseSemanticProbabilitylossNaturalnessRewardModelsingle-viewreconstructioncross-modeltransferability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that current vision-language models are not ready for the 3D variations that occur in real-world scenes, and that this failure can be demonstrated with physically reproducible images, not just digital artifacts. To do this it introduces AdvDreamer, which starts from a single natural photo, reconstructs the main object in 3D, applies an optimized rotation, translation, and scaling, and recomposes the result into the scene while a learned naturalness filter blocks corrupted outputs. The resulting adversarial 3D-transformation samples cut zero-shot classification accuracy of OpenCLIP and BLIP-2 by 65-80%, and the attacks transfer across model architectures and tasks, including commercial systems such as GPT-4o. The same samples, plus physical reproductions from video frames, form MM3DTBench, a 215-sample benchmark on which most of 13 evaluated VLMs fall below 50% accuracy. If the claim holds, VLM pretraining carries a systematic 3D bias that matters for any deployment where objects appear in novel poses.

What carries the argument

The machinery is an adversarial distribution over rigid 3D transformations, $p(\Theta)$ with $\Theta=\{\alpha,\beta,\gamma,\Delta x,\Delta y,s\}$ (yaw, pitch, roll, translations, scale), optimized by a query-based CMA-ES loop. Three modules make the loop work. The Monocular Pose Manipulation pipeline realizes $T(\Theta, X)$: Grounded-SAM cuts the image into foreground and background, TripoSR builds a single-image 3D reconstruction of the foreground, the sampled transform is applied and re-rendered, and a diffusion composition model (AnyDoor with consistent-light-transport harmonization) blends the transformed object back into the scene. The Naturalness Reward Model, a DINOv2 backbone with two small MLP heads predicting visual fidelity and physical plausibility scores, is added as a regularizer so optimization does not slide into hallucinated or unnatural images. The Inverse Semantic Probability loss, negative log-likelihood of the ground-truth label after a softmax over image-text cosine similarities, is the task- and architecture-agnostic objective that makes the attack transfer.

What would settle it

Re-run the zero-shot classification on physically captured objects manipulated to exactly the poses AdvDreamer selects, with poses verified by an independent pose estimator; if accuracy does not drop, or if it drops equally for poses that human judges consider natural, then the degradation is not inherently caused by 3D variation.

Watch

Extended reading notes

Core claim

AdvDreamer's central discovery is that worst-case 3D variations—object rotations, translations, and scalings—form a largely unexplored adversarial subspace for vision-language models, and that this subspace can be attacked from a single image using generative 3D priors. The framework optimizes a multivariate Gaussian distribution over six transformation parameters using CMA-ES, guided by two losses: the Inverse Semantic Probability loss, which pushes the ground-truth label's image-text matching probability down in the shared visual-textual embedding space, and a Naturalness Reward Model built on DINOv2, which keeps samples visually faithful and physically plausible. On ImageNet zero-shot classification, the strongest samples drive OpenCLIP and BLIP-2 accuracy from roughly 80-98% down to 15-19% (65-80% degradation), and the attacks transfer to other encoders and to captioning and VQA models. Physical video replications of the discovered poses still degrade OpenCLIP accuracy from 100% to 51.3% and LLaVa VQA accuracy to 33.6%, which the authors take as evidence that 3D variation itself, not image corruption, drives the failures. The same samples define MM3DTBench, a 215-sample benchmark where most of 13 evaluated VLMs score below 50%.

Load-bearing premise

The weakest load-bearing premise is that Monocular Pose Manipulation actually realizes the requested rigid 3D transformation—that the rendered object really is at the sampled pose $\Theta$, with no identity-changing texture or geometry corruption—since the paper never quantifies pose accuracy on generated samples and its naturalness filter agrees with humans only 67.8% of the time.

Editorial extensions

If this is right

  • Zero-shot classification accuracy of the OpenCLIP family drops by 65-80% on Adv-3DT samples, with the distribution-center sample $\Theta^*$ being the most aggressive.
  • Attacks optimized on one visual encoder transfer to other architectures, so the vulnerability is shared rather than model-specific.
  • Captioning and VQA degrade by up to roughly 50% even for GPT-4o, and physical reproductions still reduce accuracy, so the threat survives the sim-to-real gap.
  • MM3DTBench gives a 215-sample benchmark with candidate answers and orientation annotations for evaluating VLM behavior under challenging 3D variations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the 3D-bias explanation is right, adding adversarially transformed samples to VLM pretraining or instruction tuning would be a direct mitigation; the paper does not pursue training, only evaluation.
  • Because the Naturalness Reward Model agrees with human judges only 67.8% of the time, a stricter naturalness filter would likely shrink the reported digital accuracy drops, since some optimized samples may contain artifacts the reward model misses.
  • The digital-to-physical gap (100% to 0% accuracy digitally versus 100% to 51.3% physically) suggests that part of the digital attack success depends on rendering artifacts; a controlled test with real objects at exactly the discovered poses would separate pose effects from texture effects.
  • The same single-view pipeline could be turned into a 3D data-augmentation engine for robustness training, a step beyond this paper's evaluation-only scope.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces AdvDreamer, a framework that turns a single-view image into adversarial 3D transformation (Adv-3DT) samples by optimizing a distribution over 3D pose parameters via CMA-ES. The transformation is implemented by the Monocular Pose Manipulation pipeline (Grounded-SAM segmentation, TripoSR reconstruction/rendering, and diffusion-based recomposition), and is regularized by a Naturalness Reward Model trained on GPT-4o/volunteer scores. The adversarial objective is the Inverse Semantic Probability loss. The authors report large accuracy drops on OpenCLIP and BLIP models, transferability across VLMs, degradation in captioning and VQA for open-source and commercial models, and a physical reproduction study. They claim this shows current VLMs are not ready for real-world 3D variations and that degradation is inherently caused by 3D variation rather than image quality problems.

Significance. If the central causal claim is established, this is a significant contribution: a zero-shot, single-view pipeline for generating physically realizable adversarial 3D variations, an extensive evaluation across architectures and tasks, and a reusable benchmark. The breadth of victim models, the transferability analysis, and the physical-world reproduction are notable strengths. However, the causal interpretation relies on an unmeasured premise about pose fidelity, and the naturalness loop has a training/evaluation circularity, so the significance is conditional on additional verification.

major comments (3)
  1. [Sec. 3.2.1, Eq. (2); RQ3 (Sec. 4.2)] The claim that degradation is 'inherently caused by the 3D variation rather than undesirable image quality problems' is load-bearing but never directly tested. The transformation T(Θ,X)=C_{w1}(R_{w0}(X_f,Θ),X_b) is assumed to realize the specified rotation/translation/scaling, but no pose estimation error is reported for the generated Adv-3DT samples, and neither the NRM filter nor the qualitative figures measure realized angles. Because TripoSR reconstruction and diffusion recomposition can alter identity, texture, or geometry, the paper should report a pose fidelity metric (e.g., predicted object pose vs. requested Θ, or manual angle annotations) on the generated samples before interpreting the accuracy drops as purely 3D-pose effects.
  2. [Table 5, Sec. 4.2 (RQ3)] The physical experiment does not include a physical random-transformation baseline. The digital Random baseline already drops OpenCLIP accuracy from 98.0% to 62.6% (Table 1), and the physical Adv-3DT result is 51.3%; without a physical random-pose control, the physical drop may reflect generic sensitivity to pose shifts rather than the optimized adversarial pose. Adding such a control (or reporting per-object pose error) would distinguish the two explanations and close the digital-to-physical gap interpretation.
  3. [Sec. 3.2.2, Table 4, Fig. 4] The naturalness improvement claim is partly circular: the NRM is trained on GPT-4o-generated preference labels, and Table 4's ScoreR and ScoreP are assessed by GPT-4o. Moreover, the NRM agrees with human judges only 67.8% (Fig. 4), so it is a weak proxy for the visual quality that the causal claim requires. Please report human evaluation on the w/ vs. w/o NRM samples, and clarify whether the scores in Table 4 come from the NRM itself or from a separate GPT-4o judge.
minor comments (5)
  1. [Sec. 3.3, Eq. (8)] There is a typo in Eq. (8): the second 'A' in the definitions should be 'B', since B = (Θ_min + Θ_max)/2 and A = (Θ_min − Θ_max)/2.
  2. [Table 5] The table header contains the typo 'Physicial'; it should read 'Physical'.
  3. [Fig. 4] The label 'Accuracy Relative to Human' is ambiguous because the human column shows 100%; please state explicitly that human agreement is used as the reference and clarify whether GPT-4o and NRM values are agreement rates.
  4. [Table 3 and elsewhere] The metric name 'B@4' is better written as 'BLEU@4' for consistency with the text and to avoid confusion.
  5. [Sec. 3.2.1] The phrase 'neural rendering function' contains a typo; it should be 'neural rendering function' with the spelling corrected to 'neural'.

Circularity Check

1 steps flagged · score 3.0 of 10

Core vulnerability benchmark is independent, but the naturalness-improvement claim is partly self-referential: GPT-4o supervises the NRM and then scores the resulting samples.

  1. fitted input called prediction [Sec. 3.2.2 (NRM training) and Sec. 4.2 RQ3 / Table 4]
    "These scores are initially annotated automatically by GPT-4o with Chain-of-Thought prompts and verified by volunteers for correction. ... Tab. 4 presents the accuracy under OpenCLIP (ViT-B/16), alongside GPT-4o-assessed visual realism ( Score R) and physical plausibility ( Score P)."

    The Naturalness Reward Model is a regression to GPT-4o's 5-point judgments; L_Nat maximizes NRM's predicted ScoreR+ScoreP. The paper's evidence that NRM improves sample quality is Table 4, where ScoreR/ScoreP are 'GPT-4o-assessed'. Hence the evaluation oracle is the same model that generated the NRM training labels: to the extent the regressor is accurate, the optimization is rewarded for moving samples toward GPT-4o's prior scoring, and the table then re-measures with that same oracle. The human-alignment statistic (67.8%) and volunteer label verification partially break the loop, but they do not remove the by-construction coupling between the training labels and the reported evaluation metric.

full rationale

The central vulnerability claims (RQ1 and RQ2) are not circular: they are measured accuracies on external victim models (OpenCLIP, BLIP, LLaVa, GPT-4o, etc.) before and after Adv-3DT samples, with random-transformation baselines. The physical-reproduction experiment provides independent, if partial, support that at least some degradation transfers to the real world, and the digital-to-physical gap is openly reported. The MPM pipeline's pose fidelity is an untested correctness assumption, but it is not a circular derivation: the transformation is parameterized by user-specified angles rather than defined by the VLM's predictions. The admission that LISP 'can be viewed as a probabilistic variant of M Fit' is transparency about incremental novelty rather than circularity, since the loss is still evaluated on held-out victim models. The only genuine circular element is the naturalness loop: GPT-4o generates the NRM training labels, the NRM regularizes optimization, and GPT-4o then scores the resulting samples in Table 4. This makes the naturalness improvement claim partly self-referential, though human verification and the 67.8% NRM-human agreement provide a partial external anchor. Overall, the paper's main benchmark result stands independently; the circularity is confined to the quality-justification component, so a score of 3 is appropriate.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The framework rests on several externally sourced generative models (Grounded-SAM, TripoSR, AnyDoor, Stable Diffusion) whose reliability is assumed rather than measured, plus a newly trained naturalness model whose agreement with humans is modest. The optimization hyperparameters and label templates are hand-chosen. These are the main uncharged debts of the paper.

free parameters (4)
  • Transformation bounds [Theta_min, Theta_max] = Not specified numerically in main text; ranges for yaw, pitch, roll, translation, scale set following ViewFool
    Hand-chosen constraints on the 6D pose parameters, intended to preserve recognizability; they define the search space for the adversarial optimization and affect which poses are reachable.
  • CMA-ES population size K and top-k selection (k=5) = K unspecified in main text; k=5
    Hand-chosen optimizer hyperparameters controlling exploration and update; convergence studies fix iteration count at 15 but do not justify K.
  • Prompt template 'a photo of a {yi}' = Constant prompt template
    The semantic probability is computed with this fixed text template from CLIP; changing the template changes the loss surface, though this is a standard design choice.
  • Naturalness Reward Model weights = Trained on 120K GPT-4o-annotated samples
    The NRM is a learned component whose predictions are used as a regularizer in the objective; its weights encode a particular, imperfect notion of naturalness (67.8% human agreement) and affect which transformations survive optimization.
assumptions (5)
  • domain assumption Grounded-SAM segments the major instance as the target foreground object
    Sec 3.2.1 step 1; if the major instance is not the target object, the pose manipulation corrupts the wrong region and the generated sample is meaningless.
  • domain assumption TripoSR (LRM) reconstructs a 3D representation faithful enough to render specified novel poses without changing object identity
    Sec 3.2.1 step 2; the entire MPM pipeline depends on the reconstruction quality for single-view images; no quantitative pose accuracy is provided.
  • domain assumption Diffusion-based recomposition blends the transformed foreground with the background without introducing artifacts or identity changes
    Sec 3.2.1 step 3; recomposition is stochastic and can alter object appearance; the paper relies on NRM to filter failures but does not validate pose preservation.
  • domain assumption NRM's naturalness scores are a valid proxy for visual fidelity and physical plausibility
    Sec 3.2.2; NRM reaches 67.8% agreement with human labels (Fig. 4), so this proxy is noisy and could admit or reject samples for reasons unrelated to 3D pose.
  • domain assumption The label set Y (ImageNet-1K for classification, COCO-80 for captioning/VQA) is sufficient to compute the adversarial objective
    Sec 4.1; the ISP loss only covers these label lists, so the optimization may exploit omissions in the label set; transfer to other tasks relies on this choice.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AdvDreamer Unveils: Are Vision-Language Models Truly Ready for Real-World 3D Variations?." pith.science (2026). https://pith.science/paper/RZOWZ2QV

@misc{pith2026241203002,
  author       = {Pith},
  title        = {Pith review of: AdvDreamer Unveils: Are Vision-Language Models Truly Ready for Real-World 3D Variations?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RZOWZ2QV}},
  note         = {Machine review of arXiv:2412.03002}
}
read the original abstract

Vision Language Models (VLMs) have exhibited remarkable generalization capabilities, yet their robustness in dynamic real-world scenarios remains largely unexplored. To systematically evaluate VLMs' robustness to real-world 3D variations, we propose AdvDreamer, the first framework capable of generating physically reproducible Adversarial 3D Transformation (Adv-3DT) samples from single-view observations. In AdvDreamer, we integrate three key innovations: Firstly, to characterize real-world 3D variations with limited prior knowledge precisely, we design a zero-shot Monocular Pose Manipulation pipeline built upon generative 3D priors. Secondly, to ensure the visual quality of worst-case Adv-3DT samples, we propose a Naturalness Reward Model that provides continuous naturalness regularization during adversarial optimization, effectively preventing convergence to hallucinated or unnatural elements. Thirdly, to enable systematic evaluation across diverse VLM architectures and visual-language tasks, we introduce the Inverse Semantic Probability loss as the adversarial optimization objective, which solely operates in the fundamental visual-textual alignment space. Based on the captured Adv-3DT samples with high aggressiveness and transferability, we establish MM3DTBench, the first VQA benchmark dataset tailored to evaluate VLM robustness under challenging 3D variations. Extensive evaluations of representative VLMs with varying architectures reveal that real-world 3D variations can pose severe threats to model performance across various tasks.

Figures

Figures reproduced from arXiv: 2412.03002 by the authors.

Figure 1
Figure 1. The 3D Variation Vulnerabilities in VLMs. (A) The proposed AdvDreamer allow the capture of adversarial 3D transformation (Adv-3DT) samples from single natural images, threatening various VLMs. (B) These Adv-3DT samples can be successfully reproduced in the physical world and (C) retain their aggressiveness under background variations. (D) We also explain the GPT-Score metric used in the experiments. In each VLM resp… view at source ↗
Figure 2
Figure 2. Comparison of paradigms to characterize Adversarial 3D Transformation (Adv-3DT) samples. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Overview of the proposed AdvDreamer framework. To capture worst-case 3D variations in the real world, we treat rigid 3D transformations as adversarial attacks, optimizing the distribution of transformation parameters through a query-based approach. In AdvDreamer, we introduce the Monocular Pose Manipulation pipeline to perform specified 3D transformations on single-view images and guide the optimization process usin… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Architecture of NRM and computational cost and accu [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Visualization of Adv-3DT samples w/ and w/o NRM. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Attack success rate of Adv-3DT samples w.r.t. (A) opti [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

68 extracted references · 35 canonical work pages

  1. [1]

    Nocaps: Novel object caption- ing at scale

    Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Ste- fan Lee, and Peter Anderson. Nocaps: Novel object caption- ing at scale. In Proceedings of the IEEE/CVF international conference on computer vision, pages 8948–8957, 2019. 6, 7

    work page 2019

  2. [2]

    Unibench: Visual reasoning requires rethinking vision- language beyond scaling

    Haider Al-Tahan, Quentin Garrido, Randall Balestriero, Diane Bouchacourt, Caner Hazirbas, and Mark Ibrahim. Unibench: Visual reasoning requires rethinking vision- language beyond scaling. arXiv preprint arXiv:2408.04810,

    arXiv

  3. [3]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Men- sch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736,

  4. [4]

    Strike (with) a pose: Neural networks are easily fooled by strange poses of familiar objects

    Michael A Alcorn, Qi Li, Zhitao Gong, Chengfei Wang, Long Mai, Wei-Shinn Ku, and Anh Nguyen. Strike (with) a pose: Neural networks are easily fooled by strange poses of familiar objects. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 4845–4854, 2019. 2, 3, 4

    work page 2019

  5. [5]

    claude-3

    Anthropic. claude-3. https://www.anthropic.com/claude,

    work page

  6. [6]

    Synthesizing robust adversarial examples

    Anish Athalye, Logan Engstrom, Andrew Ilyas, and Kevin Kwok. Synthesizing robust adversarial examples. In Inter- national conference on machine learning , pages 284–293. PMLR, 2018. 4

    work page 2018

  7. [7]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023. 2

    arXiv 2023

  8. [8]

    Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models

    Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. Ad- vances in neural information processing systems , 32, 2019. 3

    work page 2019

Show all 68 references
  1. [9]

    Spatialvlm: Endow- ing vision-language models with spatial reasoning capabili- ties

    Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvlm: Endow- ing vision-language models with spatial reasoning capabili- ties. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 144...

  2. [10]

    Large language models are visual reasoning coordinators

    Liangyu Chen, Bo Li, Sheng Shen, Jingkang Yang, Chun- yuan Li, Kurt Keutzer, Trevor Darrell, and Ziwei Liu. Large language models are visual reasoning coordinators. Ad- vances in Neural Information Processing Systems, 36, 2024. 2

    2024

  3. [11]

    Microsoft coco captions: Data collection and evaluation server

    Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedan- tam, Saurabh Gupta, Piotr Doll ´ar, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325, 2015. 6, 7

    2015 arXiv

  4. [12]

    Anydoor: Zero-shot object-level im- age customization

    Xi Chen, Lianghua Huang, Yu Liu, Yujun Shen, Deli Zhao, and Hengshuang Zhao. Anydoor: Zero-shot object-level im- age customization. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 6593–6602, 2024. 2, 4

    2024

  5. [13]

    Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks. In Pro- ceedings of the IEEE/CVF Conference on Computer...

    2024

  6. [14]

    Reproducible scal- ing laws for contrastive language-image learning

    Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuh- mann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scal- ing laws for contrastive language-image learning. In Pro- ceedings of the IEEE/CVF Conference on Computer ...

    2023

  7. [15]

    Diffedit: Diffusion-based semantic image editing with mask guidance

    Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. Diffedit: Diffusion-based semantic image editing with mask guidance. In ICLR 2023 (Eleventh Inter- national Conference on Learning Representations), 2023. 4

    2023

  8. [16]

    On the robustness of large multimodal mod- els against image adversarial attacks

    Xuanming Cui, Alejandro Aparcedo, Young Kyun Jang, and Ser-Nam Lim. On the robustness of large multimodal mod- els against image adversarial attacks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24625–24634, 2024. 2, 3, 8

    2024

  9. [17]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. 6

    2009

  10. [18]

    Viewfool: Evaluating the ro- bustness of visual recognition to adversarial viewpoints

    Yinpeng Dong, Shouwei Ruan, Hang Su, Caixin Kang, Xingxing Wei, and Jun Zhu. Viewfool: Evaluating the ro- bustness of visual recognition to adversarial viewpoints. In Neural Information Processing Systems (NeurIPS), 2022. 2, 3, 4, 6

    2022

  11. [19]

    Google vizier: A service for black-box optimization

    Daniel Golovin, Benjamin Solnik, Subhodeep Moitra, Greg Kochanski, John Karro, and David Sculley. Google vizier: A service for black-box optimization. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1487–1495, 2017. 5

    2017

  12. [20]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Ba- tra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913...

    2017

  13. [21]

    Towards analyz- ing semantic robustness of deep neural networks

    Abdullah Hamdi and Bernard Ghanem. Towards analyz- ing semantic robustness of deep neural networks. In ECCV, pages 22–38. Springer, 2020. 2, 3, 4

    2020

  14. [22]

    The cma evolution strategy: A tutorial

    Nikolaus Hansen. The cma evolution strategy: A tutorial. arXiv preprint arXiv:1604.00772, 2016. 5, 6

    2016 arXiv

  15. [23]

    Openlrm: Open-source large reconstruction models

    Zexin He and Tengfei Wang. Openlrm: Open-source large reconstruction models. https://github.com/ 3DTopia/OpenLRM, 2023. 2

    2023

  16. [24]

    Lrm: Large reconstruction model for single image to 3d

    Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d. arXiv preprint arXiv:2311.04400, 2023. 2

    2023 arXiv

  17. [25]

    V oxposer: Composable 3d value 10 maps for robotic manipulation with language models

    Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. V oxposer: Composable 3d value 10 maps for robotic manipulation with language models. arXiv preprint arXiv:2307.05973, 2023. 2, 3

    2023 arXiv

  18. [26]

    open- clip, 2021

    Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Han- naneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. open- clip, 2021. 3, 6

    2021

  19. [27]

    Clip-benchmark

    LAION-AI. Clip-benchmark. https://github.com/LAION- AI/CLIP benchmark, 2024. 6

    2024

  20. [28]

    Mimic- it: Multi-modal in-context instruction tuning

    Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Mimic- it: Multi-modal in-context instruction tuning. arXiv preprint arXiv:2306.05425, 2023. 7

    2023 arXiv

  21. [29]

    Align before fuse: Vision and language representation learn- ing with momentum distillation

    Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learn- ing with momentum distillation. Advances in neural infor- mation processing systems, 34:9694–9705, 2021. 6

    2021

  22. [30]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In Interna- tional conference on machine learning, pages 12888–12900. PMLR, 2022. 3, 6, 7

    2022

  23. [31]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In In- ternational conference on machine learning , pages 19730– 19742. PMLR, 2023. 2, 3, 6

    2023

  24. [32]

    Visualbert: A simple and perfor- mant baseline for vision and language

    Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and perfor- mant baseline for vision and language. arXiv preprint arXiv:1908.03557, 2019. 3

    1908 arXiv

  25. [33]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceeding...

    2014

  26. [34]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024. 6

    2024

  27. [35]

    Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024. 2, 3, 7

    2024

  28. [36]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 2, 3, 6, 7

    2024

  29. [37]

    Mmbench: Is your multi-modal model an all-around player? In European Conference on Computer Vision, pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European Conference on Computer Vision, pages 216–233. Springer, 2024. 3

    2024

  30. [38]

    Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks

    Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. Advances in neural information processing systems, 32, 2019. 3

    2019

  31. [39]

    When and how con- volutional neural networks generalize to out-of-distribution category–viewpoint combinations

    Spandan Madan, Timothy Henry, Jamell Dozier, Helen Ho, Nishchal Bhandari, Tomotake Sasaki, Fr ´edo Durand, Hanspeter Pfister, and Xavier Boix. When and how con- volutional neural networks generalize to out-of-distribution category–viewpoint combinations. Nature Machine Intelli...

    2022

  32. [40]

    Understanding zero-shot adversar- ial robustness for large-scale models

    Chengzhi Mao, Scott Geng, Junfeng Yang, Xin Wang, and Carl V ondrick. Understanding zero-shot adversar- ial robustness for large-scale models. arXiv preprint arXiv:2212.07016, 2022. 3

    2022 arXiv

  33. [41]

    Nerf: Representing scenes as neural radiance fields for view syn- thesis

    Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view syn- thesis. In ECCV, pages 405–421, 2020. 3

    2020

  34. [42]

    openai. Gpt-4o. https://openai.com/index/hello-gpt-4o/,

  35. [43]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. 2, 4

    2023 arXiv

  36. [44]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  37. [45]

    Grounded sam: Assembling open-world models for diverse visual tasks

    Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, et al. Grounded sam: Assembling open-world models for diverse visual tasks. arXiv preprint arXiv:2401.14159 ,

    arXiv

  38. [46]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684–10695, 2022. 4

    2022

  39. [47]

    Towards viewpoint-invariant visual recogni- tion via adversarial training

    Shouwei Ruan, Yinpeng Dong, Hang Su, Ning Chen, and Xingxing Wei. Towards viewpoint-invariant visual recogni- tion via adversarial training. In ICCV, pages 1–10, 2023. 2, 3, 4, 6

    2023

  40. [48]

    Improving viewpoint robust- ness for visual recognition via adversarial training

    Shouwei Ruan, Yinpeng Dong, Hang Su, Jianteng Peng, Ning Chen, and Xingxing Wei. Improving viewpoint robust- ness for visual recognition via adversarial training. arXiv preprint arXiv:2307.11528, 2023. 3

    2023 arXiv

  41. [49]

    Laion-5b: An open large-scale dataset for training next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Worts- man, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural In- f...

    2022

  42. [50]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020. 5

    2010 arXiv

  43. [51]

    Eva-clip: Improved training techniques for clip at scale

    Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389, 2023. 2 11

    2023 arXiv

  44. [52]

    Drivevlm: The convergence of autonomous driving and large vision-language models

    Xiaoyu Tian, Junru Gu, Bailin Li, Yicheng Liu, Chenxu Hu, Yang Wang, Kun Zhan, Peng Jia, Xianpeng Lang, and Hang Zhao. Drivevlm: The convergence of autonomous driving and large vision-language models. arXiv preprint arXiv:2402.12289, 2024. 2

    2024 arXiv

  45. [53]

    Triposr: Fast 3d object reconstruction from a single image

    Dmitry Tochilkin, David Pankratz, Zexiang Liu, Zixuan Huang, , Adam Letts, Yangguang Li, Ding Liang, Christian Laforte, Varun Jampani, and Yan-Pei Cao. Triposr: Fast 3d object reconstruction from a single image. arXiv preprint arXiv:2403.02151, 2024. 4

    2024 arXiv

  46. [54]

    Eyes wide shut? exploring the visual shortcomings of multimodal llms, 2024

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms, 2024. 3, 5

    2024

  47. [55]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 3

    2023 arXiv

  48. [56]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023. 3

    2023 arXiv

  49. [57]

    Cogvlm: Visual expert for pretrained language models

    Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.03079, 2023. 8

    2023 arXiv

  50. [58]

    Demystifying clip data

    Hu Xu, Saining Xie, Xiaoqing Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang-Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying clip data. In The Twelfth International Conference on Learn- ing Representations, 2024. 6

    2024

  51. [59]

    Drivegpt4: Interpretable end-to-end autonomous driving via large language model

    Zhenhua Xu, Yujia Zhang, Enze Xie, Zhen Zhao, Yong Guo, Kwan-Yee K Wong, Zhenguo Li, and Hengshuang Zhao. Drivegpt4: Interpretable end-to-end autonomous driving via large language model. IEEE Robotics and Automation Let- ters, 2024. 2

    2024

  52. [60]

    Vlfm: Vision-language frontier maps for zero-shot semantic navigation

    Naoki Yokoyama, Sehoon Ha, Dhruv Batra, Jiuguang Wang, and Bernadette Bucher. Vlfm: Vision-language frontier maps for zero-shot semantic navigation. In 2024 IEEE In- ternational Conference on Robotics and Automation (ICRA), pages 42–48. IEEE, 2024. 2, 3

    2024

  53. [61]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975–11986, 2023. 2, 6

    2023

  54. [62]

    Scal- ing in-the-wild training for diffusion-based illumination har- monization and editing by imposing consistent light trans- port

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Scal- ing in-the-wild training for diffusion-based illumination har- monization and editing by imposing consistent light trans- port. In The Thirteenth International Conference on Learn- ing Representations, 2025. 2, 4

    2025

  55. [63]

    Benchmarking trustworthiness of multimodal large language models: A comprehensive study, 2024

    Yichi Zhang, Yao Huang, Yitong Sun, Chang Liu, Zhe Zhao, Zhengwei Fang, Yifan Wang, Huanran Chen, Xiao Yang, Xingxing Wei, Hang Su, Yinpeng Dong, and Jun Zhu. Benchmarking trustworthiness of multimodal large language models: A comprehensive study, 2024. 2, 3, 6, 8

    2024

  56. [64]

    Lapt: Label-driven automated prompt tuning for ood detec- tion with vision-language models

    Yabin Zhang, Wenjie Zhu, Chenhang He, and Lei Zhang. Lapt: Label-driven automated prompt tuning for ood detec- tion with vision-language models. Proceedings of the euro- pean conference on computer vision (ECCV), 2024. 2

    2024

  57. [65]

    Ood-cv: A benchmark for robustness to out-of- distribution shifts of individual nuisances in natural images

    Bingchen Zhao, Shaozuo Yu, Wufei Ma, Mingxin Yu, Shenxiao Mei, Angtian Wang, Ju He, Alan Yuille, and Adam Kortylewski. Ood-cv: A benchmark for robustness to out-of- distribution shifts of individual nuisances in natural images. In European conference on computer vision, pages ...

    2022

  58. [66]

    On evaluating adversarial robustness of large vision-language models

    Yunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang, Chongx- uan Li, Ngai-Man Cheung, and Min Lin. On evaluating adversarial robustness of large vision-language models. In Thirty-seventh Conference on Neural Information Process- ing Systems, 2023. 2, 8

    2023

  59. [67]

    Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in Neural Information Processing Systems, 36:46595–46623, 2023

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in Neural Information Processing Systems, 36:46595–46623, 2023. 6

    2023

  60. [68]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 6, 7 12

    2023 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.