{"id":"02cc5dcf-65b0-46dd-b7d6-a40d66bbf89c","arxiv_id":"2412.03002","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A single-image pipeline generates adversarial 3D pose variations that degrade vision-language model accuracy by up to 80%, and a new benchmark, MM3DTBench, quantifies this vulnerability.","lead":"This paper introduces AdvDreamer, a pipeline that generates adversarial 3D pose variations of an object from a single photo, and uses it to show that vision-language models like GPT-4o drop sharply in accuracy when objects appear in unusual poses. The tool and the new benchmark MM3DTBench could help stress-test VLMs before deployment in robots, cars, and other safety-critical systems.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The MPM pipeline's pose fidelity is never measured; if the requested angles are not faithfully realized, the claim that degradation is 'inherently caused by 3D variation' is unsupported.","rationale":"The paper is valuable and its broad conclusion—current VLMs are fragile under real-world 3D pose changes—is supported by the physical experiments and by consistent accuracy drops across models and tasks. The optimization framework with CMA-ES and the ISP loss is clearly described, and Table 5 is real evidence that something physical degrades these models. However, the strongest phrasing in RQ3 is an attribution claim, and that attribution depends on a chain of generative components (TripoSR reconstruction, AnyDoor/IC-Light composition, NRM filtering) whose combined effect on realized pose is never measured. The reader's weakest assumption identifies this exact gap, and I agree with it. One could argue that the physical experiment independently validates causality, but the absence of a random-physical-pose baseline means generic pose sensitivity is not excluded; digital random transformations already incur a 35-point drop. The concern is not that the paper is internally inconsistent, but that the central causal statement is ahead of its evidence. A pose-fidelity audit with independent re-rendering is feasible and would settle it. No code or data release currently makes this audit possible, which raises the verification cost but is not itself the logical flaw. The verdict remains conditional: accept with the requirement to quantify realized pose error and to compare physical optimized poses against a physical random baseline.","tokens_in":19812,"tokens_out":6376,"duration_ms":68593,"concrete_test":"Perform a pose-fidelity audit on 100 Adv-3DT Θ* samples by estimating the realized object pose in the composed output with an independent 6D pose estimator (e.g., MegaPose) or keypoint alignment, and compare with requested (α, β, γ, Δx, Δy, s). For the 20 samples with the largest pose error, re-render the same object at the requested pose using a CAD model or a multi-view NeRF with matched background and lighting, then re-run the Table 1 evaluation. The RQ3 claim is settled if pose-faithful re-renderings reproduce a comparable accuracy drop; if not, the digital degradation is driven by reconstruction or composition artifacts rather than pose alone.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing premise of RQ3 is that AdvDreamer's digital samples are cleanly parameterized by 3D pose. The transformation T(Θ,X) = C_{w1}(R_{w0}(X_f,Θ), X_b) assumes that rotating TripoSR's single-view reconstruction by α actually yields an object whose pose is α, with no identity or texture drift. This is not verified anywhere in the paper: no pose error metric is reported for generated samples, and neither the NRM nor the qualitative figures measures realized angles. The quality gate is weak for this purpose: NRM is trained on MPM-generated samples scored by GPT-4o and volunteers, and it agrees with human judges only 67.8% (Fig. 4); a high naturalness score does not certify that the requested pose was realized. Table 5 gives partial external support (physical reproduction degrades OpenCLIP from 100% to 51.3%), but no physical random-pose baseline is reported, so this drop may reflect generic sensitivity to pose changes—the digital Random baseline already drops to 62.6% (Table 1)—rather than the optimized adversarial pose. The digital-to-physical gap (100→0 vs 100→51.3) is consistent with digital samples containing non-reproducible reconstruction or recomposition artifacts. Therefore the specific claim that degradation is 'inherently caused by the 3D variation rather than undesirable image quality problems' is under-supported, even though the broader claim that VLMs are vulnerable to 3D variations remains plausible.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AdvDreamer, a framework that turns a single-view image into adversarial 3D transformation (Adv-3DT) samples by optimizing a distribution over 3D pose parameters via CMA-ES. The transformation is implemented by the Monocular Pose Manipulation pipeline (Grounded-SAM segmentation, TripoSR reconstruction/rendering, and diffusion-based recomposition), and is regularized by a Naturalness Reward Model trained on GPT-4o/volunteer scores. The adversarial objective is the Inverse Semantic Probability loss. The authors report large accuracy drops on OpenCLIP and BLIP models, transferability across VLMs, degradation in captioning and VQA for open-source and commercial models, and a physical reproduction study. They claim this shows current VLMs are not ready for real-world 3D variations and that degradation is inherently caused by 3D variation rather than image quality problems.","tokens_in":20179,"tokens_out":4298,"duration_ms":38971,"significance":"If the central causal claim is established, this is a significant contribution: a zero-shot, single-view pipeline for generating physically realizable adversarial 3D variations, an extensive evaluation across architectures and tasks, and a reusable benchmark. The breadth of victim models, the transferability analysis, and the physical-world reproduction are notable strengths. However, the causal interpretation relies on an unmeasured premise about pose fidelity, and the naturalness loop has a training/evaluation circularity, so the significance is conditional on additional verification.","major_comments":[{"comment":"The claim that degradation is 'inherently caused by the 3D variation rather than undesirable image quality problems' is load-bearing but never directly tested. The transformation T(Θ,X)=C_{w1}(R_{w0}(X_f,Θ),X_b) is assumed to realize the specified rotation/translation/scaling, but no pose estimation error is reported for the generated Adv-3DT samples, and neither the NRM filter nor the qualitative figures measure realized angles. Because TripoSR reconstruction and diffusion recomposition can alter identity, texture, or geometry, the paper should report a pose fidelity metric (e.g., predicted object pose vs. requested Θ, or manual angle annotations) on the generated samples before interpreting the accuracy drops as purely 3D-pose effects.","section":"Sec. 3.2.1, Eq. (2); RQ3 (Sec. 4.2)"},{"comment":"The physical experiment does not include a physical random-transformation baseline. The digital Random baseline already drops OpenCLIP accuracy from 98.0% to 62.6% (Table 1), and the physical Adv-3DT result is 51.3%; without a physical random-pose control, the physical drop may reflect generic sensitivity to pose shifts rather than the optimized adversarial pose. Adding such a control (or reporting per-object pose error) would distinguish the two explanations and close the digital-to-physical gap interpretation.","section":"Table 5, Sec. 4.2 (RQ3)"},{"comment":"The naturalness improvement claim is partly circular: the NRM is trained on GPT-4o-generated preference labels, and Table 4's ScoreR and ScoreP are assessed by GPT-4o. Moreover, the NRM agrees with human judges only 67.8% (Fig. 4), so it is a weak proxy for the visual quality that the causal claim requires. Please report human evaluation on the w/ vs. w/o NRM samples, and clarify whether the scores in Table 4 come from the NRM itself or from a separate GPT-4o judge.","section":"Sec. 3.2.2, Table 4, Fig. 4"}],"minor_comments":[{"comment":"There is a typo in Eq. (8): the second 'A' in the definitions should be 'B', since B = (Θ_min + Θ_max)/2 and A = (Θ_min − Θ_max)/2.","section":"Sec. 3.3, Eq. (8)"},{"comment":"The table header contains the typo 'Physicial'; it should read 'Physical'.","section":"Table 5"},{"comment":"The label 'Accuracy Relative to Human' is ambiguous because the human column shows 100%; please state explicitly that human agreement is used as the reference and clarify whether GPT-4o and NRM values are agreement rates.","section":"Fig. 4"},{"comment":"The metric name 'B@4' is better written as 'BLEU@4' for consistency with the text and to avoid confusion.","section":"Table 3 and elsewhere"},{"comment":"The phrase 'neural rendering function' contains a typo; it should be 'neural rendering function' with the spelling corrected to 'neural'.","section":"Sec. 3.2.1"}],"recommendation":"major_revision","confidential_remarks":"The core vulnerability measurement is extensive and likely publishable, but the causal claim in RQ3 needs substantial additional evidence. If pose fidelity cannot be verified, the authors should soften the 'inherently caused' claim to 'consistent with' 3D-pose sensitivity. The benchmark is valuable, and the naturalness circularity should be fixed. I would support acceptance after major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hi [Colleague],\n\nThe short version: this paper has a genuinely new framework and a lot of careful evaluation, but its strongest claim—that the measured degradation is caused by 3D pose rather than image artifacts—is not backed by the experiments. The vulnerability numbers are probably real, but the causal attribution is a step ahead of the evidence.\n\nWhat's new and good: AdvDreamer combines single-image 3D reconstruction (TripoSR), diffusion-based recomposition, and a learned naturalness reward in a black-box optimization loop. That is a real departure from NeRF-based or synthetic-asset approaches. The benchmark MM3DTBench, covering classification, captioning, VQA, and physical reproduction, is a useful addition. The evaluation is broad: several model families, transfer studies, ablation of the loss, and a physical reproduction with thousands of frames. The internal consistency of the accuracy numbers is good, and the Random-versus-adversarial comparison shows the optimization is doing something.\n\nSoft spots, in order of size. First, pose fidelity is never measured. The pipeline rotates a TripoSR reconstruction and expects the final image to realize the requested angle, but no pose error is reported on generated images. The recomposition step can change identity, texture, and even object shape, so the gap between requested and realized pose is unknown. Without that measurement, the paper's 'inherently caused by 3D variation' claim rests on assertion. Second, the physical experiment lacks a random-pose baseline. The physical drop (100% to 51.3%) is substantial, but the digital Random baseline already drops to 62.6%, so the physical drop may just be generic pose sensitivity, not the optimized adversarial pose. Third, the naturalness loop is partly circular: GPT-4o generates NRM training labels and GPT-4o is used to score the samples in Table 4. NRM's agreement with humans is 67.8%, which makes it a weak gate for certifying that samples are artifact-free. Finally, no code or data is released, which limits the benchmark's immediate usefulness.\n\nNone of these are fatal to the broad finding that current VLMs handle 3D pose variations poorly. That is consistent with the Random baselines and with prior work on image classifiers. But the specific contribution—adversarial pose optimization from a single view—needs direct pose measurement and a non-optimized physical control before it is fully supported.\n\nMy recommendation: this deserves peer review. A serious referee can push for pose-fidelity validation and a physical random baseline; those revisions would turn a plausible paper into a solid one.\n\nBest.","headline":"A novel single-image adversarial 3D pose pipeline with broad VLM evaluation, but the causal claim that degradation is inherently caused by 3D pose is not yet backed by pose-fidelity measurements.","tokens_in":20689,"tokens_out":2570,"would_cite":false,"duration_ms":22932,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AdvDreamer generates physically reproducible adversarial 3D transformations from single images, and current vision-language models—including GPT-4o—suffer large transferable accuracy drops on them.","keywords":["adversarial 3D transformations","vision-language models","MM3DTBench","monocular pose manipulation","Inverse Semantic Probability loss","Naturalness Reward Model","single-view 3D reconstruction","cross-model transferability"],"falsifier":"Re-run the zero-shot classification on physically captured objects manipulated to exactly the poses AdvDreamer selects, with poses verified by an independent pose estimator; if accuracy does not drop, or if it drops equally for poses that human judges consider natural, then the degradation is not inherently caused by 3D variation.","tokens_in":19658,"feed_emoji":"🔄","tokens_out":9598,"duration_ms":85272,"temperature":0.7,"pith_summary":"The paper sets out to show that current vision-language models are not ready for the 3D variations that occur in real-world scenes, and that this failure can be demonstrated with physically reproducible images, not just digital artifacts. To do this it introduces AdvDreamer, which starts from a single natural photo, reconstructs the main object in 3D, applies an optimized rotation, translation, and scaling, and recomposes the result into the scene while a learned naturalness filter blocks corrupted outputs. The resulting adversarial 3D-transformation samples cut zero-shot classification accuracy of OpenCLIP and BLIP-2 by 65-80%, and the attacks transfer across model architectures and tasks, including commercial systems such as GPT-4o. The same samples, plus physical reproductions from video frames, form MM3DTBench, a 215-sample benchmark on which most of 13 evaluated VLMs fall below 50% accuracy. If the claim holds, VLM pretraining carries a systematic 3D bias that matters for any deployment where objects appear in novel poses.","feed_headline":"VLMs drop 65-80 percent on adversarial 3D poses","feed_subtitle":"A single-view pipeline creates physically reproducible 3D variations that transfer across models and tasks.","key_machinery":"The machinery is an adversarial distribution over rigid 3D transformations, $p(\\Theta)$ with $\\Theta=\\{\\alpha,\\beta,\\gamma,\\Delta x,\\Delta y,s\\}$ (yaw, pitch, roll, translations, scale), optimized by a query-based CMA-ES loop. Three modules make the loop work. The Monocular Pose Manipulation pipeline realizes $T(\\Theta, X)$: Grounded-SAM cuts the image into foreground and background, TripoSR builds a single-image 3D reconstruction of the foreground, the sampled transform is applied and re-rendered, and a diffusion composition model (AnyDoor with consistent-light-transport harmonization) blends the transformed object back into the scene. The Naturalness Reward Model, a DINOv2 backbone with two small MLP heads predicting visual fidelity and physical plausibility scores, is added as a regularizer so optimization does not slide into hallucinated or unnatural images. The Inverse Semantic Probability loss, negative log-likelihood of the ground-truth label after a softmax over image-text cosine similarities, is the task- and architecture-agnostic objective that makes the attack transfer.","core_discovery":"AdvDreamer's central discovery is that worst-case 3D variations—object rotations, translations, and scalings—form a largely unexplored adversarial subspace for vision-language models, and that this subspace can be attacked from a single image using generative 3D priors. The framework optimizes a multivariate Gaussian distribution over six transformation parameters using CMA-ES, guided by two losses: the Inverse Semantic Probability loss, which pushes the ground-truth label's image-text matching probability down in the shared visual-textual embedding space, and a Naturalness Reward Model built on DINOv2, which keeps samples visually faithful and physically plausible. On ImageNet zero-shot classification, the strongest samples drive OpenCLIP and BLIP-2 accuracy from roughly 80-98% down to 15-19% (65-80% degradation), and the attacks transfer to other encoders and to captioning and VQA models. Physical video replications of the discovered poses still degrade OpenCLIP accuracy from 100% to 51.3% and LLaVa VQA accuracy to 33.6%, which the authors take as evidence that 3D variation itself, not image corruption, drives the failures. The same samples define MM3DTBench, a 215-sample benchmark where most of 13 evaluated VLMs score below 50%.","pith_inferences":["If the 3D-bias explanation is right, adding adversarially transformed samples to VLM pretraining or instruction tuning would be a direct mitigation; the paper does not pursue training, only evaluation.","Because the Naturalness Reward Model agrees with human judges only 67.8% of the time, a stricter naturalness filter would likely shrink the reported digital accuracy drops, since some optimized samples may contain artifacts the reward model misses.","The digital-to-physical gap (100% to 0% accuracy digitally versus 100% to 51.3% physically) suggests that part of the digital attack success depends on rendering artifacts; a controlled test with real objects at exactly the discovered poses would separate pose effects from texture effects.","The same single-view pipeline could be turned into a 3D data-augmentation engine for robustness training, a step beyond this paper's evaluation-only scope."],"forward_implications":["Zero-shot classification accuracy of the OpenCLIP family drops by 65-80% on Adv-3DT samples, with the distribution-center sample $\\Theta^*$ being the most aggressive.","Attacks optimized on one visual encoder transfer to other architectures, so the vulnerability is shared rather than model-specific.","Captioning and VQA degrade by up to roughly 50% even for GPT-4o, and physical reproductions still reduce accuracy, so the threat survives the sim-to-real gap.","MM3DTBench gives a 215-sample benchmark with candidate answers and orientation annotations for evaluating VLM behavior under challenging 3D variations."],"supporting_citations":[{"why":"Supplies the single-image 3D reconstruction prior that the Monocular Pose Manipulation pipeline rotates to new poses.","marker":"[53]"},{"why":"Provides the diffusion-based recomposition model that blends the transformed foreground back into the background.","marker":"[12]"},{"why":"Grounded-SAM performs the foreground-background decomposition that feeds the pose manipulation pipeline.","marker":"[45]"},{"why":"DINOv2 is the visual backbone of the Naturalness Reward Model that regularizes sample quality.","marker":"[43]"},{"why":"ViewFool establishes the adversarial-viewpoint formulation and distribution-over-poses optimization that AdvDreamer extends to single-view generative settings.","marker":"[18]"},{"why":"Supplies the CMA-ES black-box optimizer used to update the Gaussian transformation distribution.","marker":"[22]"},{"why":"ImageNet is the test set for the zero-shot classification experiments that produce the headline 65-80% degradation figures.","marker":"[17]"},{"why":"OpenCLIP is a primary victim model and also serves as the surrogate encoder for optimizing transfer attacks.","marker":"[26]"},{"why":"BLIP-2 is a primary victim model whose 65-80% classification degradation anchors the central claim.","marker":"[31]"},{"why":"Supplies the 'a photo of a {yi}' prompt template used by the Inverse Semantic Probability loss and the OpenAI CLIP encoder used as a surrogate for attacking closed models.","marker":"[44]"}],"fun_headline_variants":["3D pose attacks drop VLM accuracy by up to 80%","Single-view 3D attacks expose VLM blind spots","AdvDreamer turns 3D variations into VLM threats","Real-world 3D shifts fool top vision-language models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The weakest load-bearing premise is that Monocular Pose Manipulation actually realizes the requested rigid 3D transformation—that the rendered object really is at the sampled pose $\\Theta$, with no identity-changing texture or geometry corruption—since the paper never quantifies pose accuracy on generated samples and its naturalness filter agrees with humans only 67.8% of the time.","fun_headline_variants_meta":{"raw":{"variants":["3D pose attacks drop VLM accuracy by up to 80%","Single-view 3D attacks expose VLM blind spots","AdvDreamer turns 3D variations into VLM threats","Real-world 3D shifts fool top vision-language models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000184,"raw_usage":{"total_tokens":1376,"prompt_tokens":1064,"completion_tokens":312,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":680,"completion_tokens_details":{"reasoning_tokens":241}},"tokens_in":680,"tokens_out":312,"duration_ms":3862,"temperature":1.0,"reasoning_tokens":241,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:52:08.511002+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the zero-shot classification on physically captured objects manipulated to exactly the poses AdvDreamer selects, with poses verified by an independent pose estimator; if accuracy does not drop, or if it drops equally for poses that human judges consider natural, then the degradation is not inherently caused by 3D variation.","supporting_citations":[{"cited_title":"Anydoor: Zero-shot object-level im- age customization","cited_arxiv_id":null,"evidence_quote":"Provides the diffusion-based recomposition model that blends the transformed foreground back into the background."},{"cited_title":"Viewfool: Evaluating the ro- bustness of visual recognition to adversarial viewpoints","cited_arxiv_id":null,"evidence_quote":"ViewFool establishes the adversarial-viewpoint formulation and distribution-over-poses optimization that AdvDreamer extends to single-view generative settings."},{"cited_title":"open- clip, 2021","cited_arxiv_id":null,"evidence_quote":"OpenCLIP is a primary victim model and also serves as the surrogate encoder for optimizing transfer attacks."},{"cited_title":"Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models","cited_arxiv_id":null,"evidence_quote":"BLIP-2 is a primary victim model whose 65-80% classification degradation anchors the central claim."},{"cited_title":"Learning transferable visual models from natural language supervi- sion","cited_arxiv_id":null,"evidence_quote":"Supplies the 'a photo of a {yi}' prompt template used by the Inverse Semantic Probability loss and the OpenAI CLIP encoder used as a surrogate for attacking closed models."}],"review_version":1}