Pith. sign in

REVIEW 2 major objections 6 minor 72 references

Disrupting Vision-Language Model-Driven Navigation Services via Adversarial Object Fusion

T0 review · 2 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Adversarial Object Fusion (AdvOF) can craft a single 3D object that a VLM-powered navigation agent mistakes for an attacker-chosen label from multiple viewpoints, and this misperception propagates into navigation failure.

desk verdict Strong perception-level attack, but the navigation-failure claim rests on a table that violates the paper's own SPL<=SR inequality. read the letter →

arxiv 2505.23266 v1 pith:BBKLOGGG submitted 2025-05-29 cs.CR cs.AIcs.CV

Chunlong Xie , Jialing He , Shangwei Guo , Jiacheng Wang , Shudong Zhang , Tianwei Zhang , Tao Xiang This is my paper
classification cs.CRcs.AIcs.CV
keywords adversarialobjectvision-and-languagenavigationvision-languagemodelattackmulti-viewrobustnesssemanticmapembodiedAIservicesecurity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Adversarial Object Fusion (AdvOF) is an attack that creates a single 3D object able to fool the vision-language perception of navigation agents: a chair is perceived as a fire extinguisher from many viewpoints. The paper argues that this misperception propagates into the semantic map, so the agent navigates to the wrong place or fails to stop at the target. AdvOF is evaluated by rendering a bounded point-cloud perturbation into RGB-D frames and regenerating the semantic map in a simulator; it lowers success rate from 0.297 to 0.122 on Vlmaps and raises perception attack success to 0.92 while leaving normal navigation nearly unchanged. If the attack transfers from simulation to physical objects (painting or 3D printing), service robots and assistive systems with VLM-based perception would be exposed to a practical spoofing threat.

What carries the argument

The central object is the 3D perturbation $\delta_{\text{adv}}$ on the victim object's point cloud, restricted by $\|\delta_{\text{adv}}\|_p \le \epsilon$, rendered through pinhole projection into each camera view. Three mechanisms carry the argument. Aligned Object Rendering locates the victim object: an open-set VLM query, an object detector, a segmentation model, back-projection of masks into 3D, and density-based clustering to choose an object. Adversarial Collaborative Optimization defines the losses: $\mathcal{L}_{3D}$ combines color similarity and Chamfer distance for physical closeness, while $\mathcal{L}_{2D}$ combines masked image-to-image alignment, image-to-text alignment with the target label, and a background-consistency subtraction term. Adversarial Object Fusion weights each view by detection score and pixel count, applies local gradient updates, and rejects updates that break consistency with previous views, so the final object is a stable fusion across views.

What would settle it

Fabricate a physical copy of an AdvOF-optimized object and place it in a real room, then run a VLM-powered navigation agent with a real camera; if the agent does not misclassify the object as the target label from multiple views, or if navigation success does not drop measurably, the central claim of a physical-world threat fails. A cheaper in-simulator check is to render the perturbed mesh directly rather than replacing RGB-D data, and see whether the same attack success is maintained.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that a 3D adversarial object can be generated by solving a joint optimization: a geometric loss keeps the object physically close to the victim object in color and Chamfer distance, while a masked VLM feature loss pushes the rendered object's visual embedding toward the text embedding of an attacker-chosen label. The object is optimized across views with importance weights and iteratively fused, so the misclassification holds from distant and close-up, seen and unseen viewpoints. The paper reports that this object-level confusion is what breaks navigation: with the adversarial object in the scene, the four tested VLN agents (Vlmaps, Cows, CF, ORION) show lower key-point accuracy, lower success rate, and longer paths, while the same object in a normal environment barely changes agent behavior.

Load-bearing premise

The load-bearing premise is that a point-cloud perturbation optimized in the simulator and injected into RGB-D frames corresponds to a physically fabricable object, such as one made by painting or 3D printing; the paper does not fabricate a physical object or test real cameras, lighting, or sensor noise.

Editorial extensions

If this is right

  • If AdvOF works as reported, a single adversarial object can reduce the success rate of VLM-powered navigation agents by more than half (Vlmaps success rate drops from 0.297 to 0.122 in the targeted setting).
  • The perception module is the vulnerable link: attack success rate reaches 0.92 on perception, so defending navigation requires hardening the semantic map, not just the planner.
  • Because the attack transfers across image encoders, datasets, and model architectures, black-box agents are also exposed, not only agents with a known VLM.
  • The attack's resistance to shear, scaling, Gaussian noise, and brightness changes means simple image preprocessing will not reliably remove the object.
  • Attacking an early goal object disrupts later goals more than attacking a final goal, meaning the attack can compound navigation errors through the instruction sequence.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the decisive untested step is physical realizability; if painting or 3D printing cannot reproduce the optimized perturbation, the threat remains a simulation-only phenomenon.
  • Editorial inference: the same weighted multi-view optimization could be repurposed as a robustness probe for other embodied systems that build maps from VLM features, such as assistive robots with open-vocabulary grounding.
  • Editorial inference: a natural defensive extension would be multi-view consistency checks, requiring the same object to produce the same label from different viewpoints, since the attack's power comes from making diverse views agree on a false label.
  • Editorial inference: quantifying the perceptual distance between the perturbed and original object under human viewing would clarify whether the object is a subtle spoof or an obvious alteration.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper proposes Adversarial Object Fusion (AdvOF), a framework for generating 3D adversarial objects that mislead VLM-based perception modules in vision-and-language navigation (VLN) agents. The method aligns victim objects in 2D and 3D, optimizes a point-cloud perturbation using image-image and image-text embedding similarities with physical regularization, and fuses multi-view updates with importance weights. Experiments on four VLN agents (Vlmaps, Cow, CF, ORION) report strong perception-level attack success (Table 4: Acc 0.02-0.05, Asr 0.91-0.94) and large SR drops in Table 3, plus transferability and ablation studies. The paper claims the attack degrades navigation performance while preserving normal-environment behavior.

Significance. If the reported results are correct, the paper is a valuable contribution to adversarial robustness of VLM-powered embodied agents. The perception-level results are clean and represent a clear improvement over 2D and 3D baselines, and the transferability experiments (Tables 5-7) and ablation (Table 9) provide useful evidence for the method's components. However, the central navigation-failure claim is currently supported only by Table 3, which contains internally inconsistent metric values (SPL > SR under the paper's own definitions). The simulation-only evaluation also leaves the physical-world threat model unvalidated. These issues do not invalidate the perception-level contribution, but they require substantial revision before the navigation-disruption claim can be accepted.

major comments (2)
  1. [Table 3 / Section 5.1] The reported SPL values are inconsistent with the paper's own definitions. Section 5.1 defines SR as the fraction of episodes where the agent stops within 1.0 m of the target and SPL as success weighted by the oracle shortest path length normalized by the actual path length. Under these definitions SPL <= SR for every episode and hence for every aggregate. Table 3 reports, for example, Vlmaps AdvOF(Targeted) with SR=0.122 and SPL=77.5, and similar violations in every row of the attacked block (Cow SR=0.067/SPL=19.9, CF SR=0.087/SPL=27.0, ORION SR=0.064/SPL=15.6) and in the Base rows. The magnitudes (tens of meters) suggest the column actually reports average path length (PL) or a similar quantity, not the standard SPL. Since Table 3 is the only evidence that perception errors propagate to navigation failure, the main claim is unsupported until the metric is recomputed under the stated definition or the column is relabeled and the definitions are aligned.
  2. [Section 5.1, Attack Implementations] The evaluation replaces RGB-D data associated with the victim object with perturbed data and regenerates the semantic map. No physical object is fabricated and no real-world camera, lighting, or sensor noise is tested. The abstract and Section 3.2 explicitly frame the attack as a threat to physical-world deployments and list painting and 3D printing as attacker capabilities, so the simulation-to-physical transfer is a load-bearing assumption. The authors should provide a physical validation (e.g., a printed object) or explicitly scope the claimed threat to simulated environments and discuss the transfer gap.
minor comments (6)
  1. [Section 5.1] In 'Simulation Datasets and Environments', the dataset is referred to as 'HP3D' but Table 1 lists 'HM3D'; the text should be corrected to HM3D.
  2. [Equation (19)] The formula for the view weight w_v is malformed ('wv = wP v in V w , where w = score v + N v/N'); please rewrite it with a clear normalization and define N.
  3. [Algorithm 1] The loop-variable notation is garbled (e.g., 'w v v', 'maskv v', and 'delta adv wv <- delta adv v'); please fix the typesetting so the update rule is readable.
  4. [Section 4.3 / Figure 4] The text says 'Fig. 6-(d) demonstrates that AdvOF can precisely manipulate the perception of the adversarial object', but Fig. 6(d) is the ST baseline; the AdvOF result is in Fig. 6(f).
  5. [Section 5.2] There is a typo in 'optimizes adversarial objects using trajectory historis'; it should be 'trajectory histories'.
  6. [Table 3 header] The arrow directions in the Table 3 header are inconsistent with the attack goal: the 'Attacked' block has SPL up and the 'Normal' block has SPL down, which would imply that higher SPL is worse under attack and lower SPL is better in normal operation. This should be aligned with the corrected metric definition.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the central claim is a white-box attack evaluated on its own objective, which is standard attack practice, and the transfer, ablation, and noise experiments provide external checks.

full rationale

AdvOF is a white-box adversarial attack, so the observation that its evaluation (Tables 3-4) measures VLM misclassification of objects that Eqs. 16-18 were explicitly optimized to misclassify is the standard object of attack evaluation, not a circular derivation: the paper claims to construct an attack, and the evaluation verifies that constructed artifact's effect. The framework is internally self-consistent with its inputs, but its inputs (victim object, target label, VLM parameters, rendering pipeline) are external, and no parameter is fitted to the reported outcomes; the hyperparameters (alpha=0.5, beta=0.01, epsilon=32/255, mu1=0.01, mu2=0.05) are fixed settings, and the ablation (Table 9) shows each designed component contributes to the result. Genuine external checks exist: held-out-view perception (Table 4), cross-encoder transfer (Table 5), cross-dataset transfer (Table 6), cross-model transfer (Table 7), and image-noise robustness (Table 8) all confirm behavior beyond the optimized views and models, so the attack result is not forced by construction. The only self-citations are [14] and [15] (co-author Jiacheng Wang) in a background enumeration of VLM and generative-AI works; they carry no load in the derivation, and no uniqueness theorem or ansatz is imported from prior work by the same authors. Two issues are worth flagging as correctness risks rather than circularity: (1) Table 3 reports SPL values (77.5, 19.9, 27.0, 15.6 for AdvOF Targeted rows) that exceed the corresponding SR values (0.122, 0.067, 0.087, 0.064), violating the paper's own metric definitions under which SPL is bounded above by SR for every episode, so the navigation-failure measurements are internally inconsistent; (2) Section 5.1 evaluates by replacing the victim object's RGB-D data with perturbed renders and regenerating the semantic map, so physical realizability through painting or 3D printing is assumed rather than demonstrated. Neither issue constitutes a reduction of the central claim to its own inputs by definition, so the circularity score is minimal.

Assumptions & free parameters 7 free parameters · 6 assumptions · 0 invented entities

The central claim rests on standard rendering math, a set of domain assumptions about how VLM features transfer to VLN semantic maps, and an untested assumption that the simulated RGB-D replacement corresponds to a physical object modification. The attack pipeline introduces several hand-set hyperparameters but no new physical or theoretical entities.

free parameters (7)
  • epsilon (perturbation bound) = 32/255
    Upper bound on ||delta_adv||_p chosen by hand in Section 5.1; Figure 8 shows attack performance grows with larger bounds, so the result is sensitive to this choice.
  • alpha (I2T weight) = 0.5
    Balancing coefficient in Equations 15 and 18; no sensitivity analysis is reported.
  • beta (B2B weight) = 0.01
    Balancing coefficient for the background-consistency term in Equations 15 and 18; no sensitivity analysis is reported.
  • mu1 (consistency threshold) = 0.01
    Threshold in Equation 21 that rejects local updates when their mean squared error to the prior view is too large; set by hand.
  • mu2 (fusion threshold) = 0.05
    Threshold in Equation 22 that discards local updates when perception discrepancy exceeds the bound; set by hand.
  • max optimization iterations = 200
    Iteration budget in Algorithm 1 and Section 5.1; affects optimization convergence and attack success.
  • grounding box and text thresholds = 0.40
    Detection thresholds for the grounding model in Section 5.1; influence which object regions are selected as the victim.
assumptions (6)
  • standard math Pinhole camera projection (Equation 5) maps 3D object perturbations to 2D perturbations exactly.
    Used in Aligned Object Rendering; assumes calibrated intrinsics and extrinsics and no lens distortion.
  • domain assumption VLM embeddings (CLIP, LSeg, LLaVA) are differentiable surrogates for the agent's semantic map features.
    Adversarial Collaborative Optimization optimizes cosine similarity in VLM embedding space and assumes this transfers to the actual map features used by Vlmaps, Cow, CF, and ORION.
  • domain assumption Masked feature regularization leaves background semantics unchanged.
    Equation 14 assumes the background-consistency term preserves predictions outside the object mask, which Figure 4 illustrates but does not prove generally.
  • domain assumption A 3D point-cloud perturbation plus rendering faithfully represents a physically realizable adversarial object.
    Section 5.1 replaces RGB-D data with perturbed data; no physical object fabrication or real-world test is performed.
  • domain assumption Grounding DINO, SAM, and DBSCAN correctly isolate the victim object in 3D.
    Equations 2 to 4; the ablation in Table 9 shows removing alignment degrades attack performance, so this assumption is load-bearing.
  • ad hoc to paper View importance weights based on grounding score and pixel count improve optimization.
    Equation 19 defines the heuristic weighting; no theoretical justification is given beyond the motivation that close or complete views matter more.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Disrupting Vision-Language Model-Driven Navigation Services via Adversarial Object Fusion." pith.science (2026). https://pith.science/paper/BBKLOGGG

@misc{pith2026250523266,
  author       = {Pith},
  title        = {Pith review of: Disrupting Vision-Language Model-Driven Navigation Services via Adversarial Object Fusion},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BBKLOGGG}},
  note         = {Machine review of arXiv:2505.23266}
}
read the original abstract

We present Adversarial Object Fusion (AdvOF), a novel attack framework targeting vision-and-language navigation (VLN) agents in service-oriented environments by generating adversarial 3D objects. While foundational models like Large Language Models (LLMs) and Vision Language Models (VLMs) have enhanced service-oriented navigation systems through improved perception and decision-making, their integration introduces vulnerabilities in mission-critical service workflows. Existing adversarial attacks fail to address service computing contexts, where reliability and quality-of-service (QoS) are paramount. We utilize AdvOF to investigate and explore the impact of adversarial environments on the VLM-based perception module of VLN agents. In particular, AdvOF first precisely aggregates and aligns the victim object positions in both 2D and 3D space, defining and rendering adversarial objects. Then, we collaboratively optimize the adversarial object with regularization between the adversarial and victim object across physical properties and VLM perceptions. Through assigning importance weights to varying views, the optimization is processed stably and multi-viewedly by iterative fusions from local updates and justifications. Our extensive evaluations demonstrate AdvOF can effectively degrade agent performance under adversarial conditions while maintaining minimal interference with normal navigation tasks. This work advances the understanding of service security in VLM-powered navigation systems, providing computational foundations for robust service composition in physical-world deployments.

Figures

Figures reproduced from arXiv: 2505.23266 by the authors.

Figure 1
Figure 1. Attacking a VLN agent with the adversarial object. The VLN [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Vision-and-Language navigation agents with adversarial environment. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Pipeline for generating adversarial environment towards VLN agents: It optimizes the adversarial object, transforming a normal environment [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Visualization of optimization results under different regularization terms for the target attack (table [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: The scene displayed on the habitat simulator. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: The adversarial examples visualization of different attacks (chair [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: The adversarial object of different views. TV(black) [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Attacking results of different upper bounds. [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]
Figure 9
Figure 9. Figure 9: The attacking results of the adversarial object against to different noise. fridge (black) [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]
Figure 10
Figure 10. Figure 10: Attacking results of different subordinate goals. [PITH_FULL_IMAGE:figures/full_fig_p011_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

72 extracted references · 56 canonical work pages

  1. [1]

    Efficient qos-aware service recommendation for multi-tenant service-based systems in cloud,

    Y. Wang, Q. He, X. Zhang, D. Ye, and Y. Yang, “Efficient qos-aware service recommendation for multi-tenant service-based systems in cloud,”IEEE Transactions on Services Computing, vol. 13, no. 6, pp. 1045–1058, 2017

    work page 2017

  2. [2]

    Edge computing for internet of everything: A survey,

    X. Kong, Y. Wu, H. Wang, and F. Xia, “Edge computing for internet of everything: A survey,”IEEE Internet of Things Journal, vol. 9, no. 23, pp. 23 472–23 485, 2022

    work page 2022

  3. [3]

    Groupchain: Towards a scalable public blockchain in fog computing of iot services com- puting,

    K. Lei, M. Du, J. Huang, and T. Jin, “Groupchain: Towards a scalable public blockchain in fog computing of iot services com- puting,”IEEE Transactions on Services Computing, vol. 13, no. 2, pp. 252–262, 2020

    work page 2020

  4. [4]

    Vision-and-language navigation: Interpreting visually-grounded navigation instruc- tions in real environments,

    P . Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. S¨underhauf, I. Reid, S. Gould, and A. Van Den Hengel, “Vision-and-language navigation: Interpreting visually-grounded navigation instruc- tions in real environments,” inProceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 3674–3683

    work page 2018

  5. [5]

    Transferable representation learning in vision-and- language navigation,

    H. Huang, V . Jain, H. Mehta, A. Ku, G. Magalhaes, J. Baldridge, and E. Ie, “Transferable representation learning in vision-and- language navigation,” inProceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 7404–7413

    work page 2019

  6. [6]

    Look before you leap: Bridging model-free and model-based reinforcement learning for planned-ahead vision-and-language navigation,

    X. Wang, W. Xiong, H. Wang, and W. Y. Wang, “Look before you leap: Bridging model-free and model-based reinforcement learning for planned-ahead vision-and-language navigation,” in Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 37–53

    work page 2018

  7. [7]

    Reinforced cross-modal matching and self-supervised imitation learning for vision-language navi- gation,

    X. Wang, Q. Huang, A. Celikyilmaz, J. Gao, D. Shen, Y.-F. Wang, W. Y. Wang, and L. Zhang, “Reinforced cross-modal matching and self-supervised imitation learning for vision-language navi- gation,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 6629–6638

    work page 2019

  8. [8]

    Lan- guage models are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P . Dhari- wal, A. Neelakantan, P . Shyam, G. Sastry, A. Askellet al., “Lan- guage models are few-shot learners,”Advances in neural informa- tion processing systems, vol. 33, pp. 1877–1901, 2020

    1901

Show all 72 references
  1. [9]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkatet al., “Gpt-4 technical report,”arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  2. [10]

    When search engine services meet large language models: visions and challenges,

    H. Xiong, J. Bian, Y. Li, X. Li, M. Du, S. Wang, D. Yin, and S. Helal, “When search engine services meet large language models: visions and challenges,”IEEE Transactions on Services Computing, 2024

    2024

  3. [11]

    Learning transfer- able visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P . Mishkin, J. Clarket al., “Learning transfer- able visual models from natural language supervision,” inICML. PMLR, 2021, pp. 8748–8763

    2021

  4. [12]

    Visual instruction tuning,

    H. Liu, C. Li, Q. Wu, and Y. J. Lee, “Visual instruction tuning,” Advances in neural information processing systems, vol. 36, pp. 34 892– 34 916, 2023

    2023

  5. [13]

    Jarvis: Joining adversarial training with vision transformers in next-activity prediction,

    V . Pasquadibisceglie, A. Appice, G. Castellano, and D. Malerba, “Jarvis: Joining adversarial training with vision transformers in next-activity prediction,”IEEE Transactions on Services Computing, vol. 17, no. 4, pp. 1593–1606, 2023

    2023

  6. [14]

    Generative ai based secure wireless sensing for isac networks,

    J. Wang, H. Du, Y. Liu, G. Sun, D. Niyato, S. Mao, D. I. Kim, and X. Shen, “Generative ai based secure wireless sensing for isac networks,”arXiv preprint arXiv:2408.11398, 2024

    2024 arXiv

  7. [15]

    Multi-objective aerial collaborative secure communication opti- mization via generative diffusion model-enabled deep reinforce- ment learning,

    C. Zhang, G. Sun, J. Li, Q. Wu, J. Wang, D. Niyato, and Y. Liu, “Multi-objective aerial collaborative secure communication opti- mization via generative diffusion model-enabled deep reinforce- ment learning,”IEEE Transactions on Mobile Computing, 2024

    2024

  8. [16]

    Visual language maps for robot navigation,

    C. Huang, O. Mees, A. Zeng, and W. Burgard, “Visual language maps for robot navigation,” inICRA, 2023, pp. 10 608–10 615

    2023

  9. [17]

    Cows on pasture: Baselines and benchmarks for language-driven zero-shot object navigation,

    S. Y. Gadre, M. Wortsman, G. Ilharco, L. Schmidt, and S. Song, “Cows on pasture: Baselines and benchmarks for language-driven zero-shot object navigation,” inCVPR, 2023, pp. 23 171–23 181

    2023

  10. [18]

    Think, act, and ask: Open- world interactive personalized robot navigation,

    Y. Dai, R. Peng, S. Li, and J. Chai, “Think, act, and ask: Open- world interactive personalized robot navigation,” inICRA, 2024, pp. 3296–3303

    2024

  11. [19]

    CLIP-fields: Weakly supervised semantic fields for robotic mem- ory,

    N. M. M. Shafiullah, C. Paxton, L. Pinto, S. Chintala, and A. Szlam, “CLIP-fields: Weakly supervised semantic fields for robotic mem- ory,” inICRA Workshop, 2023

    2023

  12. [20]

    A survey on robotics with foundation models: toward embodied ai,

    Z. Xu, K. Wu, J. Wen, J. Li, N. Liu, Z. Che, and J. Tang, “A survey on robotics with foundation models: toward embodied ai,”arXiv preprint arXiv:2402.02385, 2024

    2024 arXiv

  13. [21]

    Real-world robot applications of foundation models: A review,

    K. Kawaharazuka, T. Matsushima, A. Gambardella, J. Guo, C. Pax- ton, and A. Zeng, “Real-world robot applications of foundation models: A review,”Advanced Robotics, pp. 1–23, 2024

    2024

  14. [22]

    Velma: Verbalization embodiment of llm agents for vision and language navigation in street view,

    R. Schumann, W. Zhu, W. Feng, T.-J. Fu, S. Riezler, and W. Y. Wang, “Velma: Verbalization embodiment of llm agents for vision and language navigation in street view,” inAAAI, vol. 38, no. 17, 2024, pp. 18 924–18 933

    2024

  15. [23]

    Segment anything,

    A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Loet al., “Segment anything,” inCVPR, 2023, pp. 4015–4026

    2023

  16. [24]

    Defending adversarial attacks via semantic feature manipulation,

    S. Wang, S. Nepal, C. Rudolph, M. Grobler, S. Chen, T. Chen, and Z. An, “Defending adversarial attacks via semantic feature manipulation,”IEEE Transactions on Services Computing, vol. 15, no. 6, pp. 3184–3197, 2021

    2021

  17. [25]

    Exploring the robustness of decision-level through adversarial attacks on llm- based embodied models,

    S. Liu, J. Chen, S. Ruan, H. Su, and Z. Yin, “Exploring the robustness of decision-level through adversarial attacks on llm- based embodied models,”arXiv preprint arXiv:2405.19802, 2024

    2024 arXiv

  18. [26]

    Poex: Policy executable embodied ai jailbreak attacks,

    X. Lu, Z. Huang, X. Li, W. Xuet al., “Poex: Policy executable embodied ai jailbreak attacks,”arXiv preprint arXiv:2412.16633, 2024

    2024 arXiv

  19. [27]

    Exploring the adversarial vulnerabili- ties of vision-language-action models in robotics,

    T. Wang, D. Liu, J. C. Liang, W. Yang, Q. Wang, C. Han, J. Luo, and R. Tang, “Exploring the adversarial vulnerabili- ties of vision-language-action models in robotics,”arXiv preprint arXiv:2411.13587, 2024

    2024 arXiv

  20. [28]

    How secure are large language models (llms) for navigation in urban environ- ments?

    C. Wen, J. Liang, S. Yuan, H. Huang, and Y. Fang, “How secure are large language models (llms) for navigation in urban environ- ments?”arXiv preprint arXiv:2402.09546, 2024

    2024 arXiv

  21. [29]

    Universal and transferable adversarial attacks on aligned language models,

    A. Zou, Z. Wang, N. Carlini, M. Nasr, J. Z. Kolter, and M. Fredrik- son, “Universal and transferable adversarial attacks on aligned language models,”arXiv preprint arXiv:2307.15043, 2023

    2023 arXiv

  22. [30]

    Accelerating greedy coordinate gradient and general prompt optimization via probe sampling,

    Y. Zhao, W. Zheng, T. Cai, X. Do Long, K. Kawaguchi, A. Goyal, and M. Q. Shieh, “Accelerating greedy coordinate gradient and general prompt optimization via probe sampling,”Advances in Neural Information Processing Systems, vol. 37, pp. 53 710–53 731, 2024

    2024

  23. [31]

    Jailbreaker: Automated jailbreak across multiple large language model chatbots,

    G. Deng, Y. Liu, Y. Li, K. Wang, Y. Zhang, Z. Li, H. Wang, T. Zhang, and Y. Liu, “Jailbreaker: Automated jailbreak across multiple large language model chatbots,” inNDSS, 2024

    2024

  24. [32]

    Tree of attacks: Jailbreaking black- box llms automatically,

    A. Mehrotra, M. Zampetakis, P . Kassianik, B. Nelson, H. Ander- son, Y. Singer, and A. Karbasi, “Tree of attacks: Jailbreaking black- box llms automatically,”Advances in Neural Information Processing Systems, vol. 37, pp. 61 065–61 105, 2024. 13

    2024

  25. [33]

    Adv- clip: Downstream-agnostic adversarial examples in multimodal contrastive learning,

    Z. Zhou, S. Hu, M. Li, H. Zhang, Y. Zhang, and H. Jin, “Adv- clip: Downstream-agnostic adversarial examples in multimodal contrastive learning,” inACMMM, 2023, pp. 6311–6320

    2023

  26. [34]

    On evaluating adversarial robustness of large vision- language models,

    Y. Zhao, T. Pang, C. Du, X. Yang, C. Li, N.-M. M. Cheung, and M. Lin, “On evaluating adversarial robustness of large vision- language models,”NeurIPS, vol. 36, 2023

    2023

  27. [35]

    Spatiotemporal attacks for embodied agents,

    A. Liu, T. Huang, X. Liu, Y. Xu, Y. Ma, X. Chen, S. J. Maybank, and D. Tao, “Spatiotemporal attacks for embodied agents,” inComputer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVII 16. Springer, 2020, pp. 122–138

    2020

  28. [36]

    Towards transferable targeted 3d adversarial attack in the physical world,

    Y. Huang, Y. Dong, S. Ruan, X. Yang, H. Su, and X. Wei, “Towards transferable targeted 3d adversarial attack in the physical world,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 24 512–24 522

    2024

  29. [37]

    Very deep convolutional networks for large-scale image recognition,

    K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,”arXiv preprint arXiv:1409.1556, 2014

    2014 arXiv

  30. [38]

    Mask r-cnn,

    K. He, G. Gkioxari, P . Doll ´ar, and R. Girshick, “Mask r-cnn,” in Proceedings of the IEEE international conference on computer vision, 2017, pp. 2961–2969

    2017

  31. [39]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” inProceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778

    2016

  32. [40]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhouet al., “Chain-of-thought prompting elicits reasoning in large language models,”NeurIPS, vol. 35, pp. 24 824–24 837, 2022

    2022

  33. [41]

    An embodied generalist agent in 3d world,

    J. Huang, S. Yong, X. Ma, X. Linghu, P . Li, Y. Wang, Q. Li, S.-C. Zhu, B. Jia, and S. Huang, “An embodied generalist agent in 3d world,” inICLR 2024 Workshop: How Far Are We From AGI, 2024. [Online]. Available: https://openreview.net/forum?id=ltX3S0juSa

    2024

  34. [42]

    Llm-planner: Few-shot grounded planning for embodied agents with large language models,

    C. H. Song, J. Wu, C. Washington, B. M. Sadler, W.-L. Chao, and Y. Su, “Llm-planner: Few-shot grounded planning for embodied agents with large language models,” inCVPR, 2023, pp. 2998– 3009

    2023

  35. [43]

    Towards learning a generalist model for embodied navigation,

    D. Zheng, S. Huang, L. Zhao, Y. Zhong, and L. Wang, “Towards learning a generalist model for embodied navigation,” inCVPR, 2024, pp. 13 624–13 634

    2024

  36. [44]

    Zson: Zero-shot object-goal navigation using multimodal goal embeddings,

    A. Majumdar, G. Aggarwal, B. Devnani, J. Hoffman, and D. Batra, “Zson: Zero-shot object-goal navigation using multimodal goal embeddings,”NeurIPS, vol. 35, pp. 32 340–32 352, 2022

    2022

  37. [45]

    A survey on large language model (llm) security and privacy: The good, the bad, and the ugly,

    Y. Yao, J. Duan, K. Xu, Y. Cai, Z. Sun, and Y. Zhang, “A survey on large language model (llm) security and privacy: The good, the bad, and the ugly,”High-Confidence Computing, p. 100211, 2024

    2024

  38. [46]

    Large language model alignment: A survey,

    T. Shen, R. Jin, Y. Huang, C. Liu, W. Dong, Z. Guo, X. Wu, Y. Liu, and D. Xiong, “Large language model alignment: A survey,”arXiv preprint arXiv:2309.15025, 2023

    2023 arXiv

  39. [47]

    Bert-attack: Adversarial attack against bert using bert,

    L. Li, R. Ma, Q. Guo, X. Xue, and X. Qiu, “Bert-attack: Adversarial attack against bert using bert,” inEMNLP, 2020, pp. 6193–6202

    2020

  40. [48]

    How robust is google’s bard to adversarial image attacks?

    Y. Dong, H. Chen, J. Chen, Z. Fang, X. Yang, Y. Zhang, Y. Tian, H. Su, and J. Zhu, “How robust is google’s bard to adversarial image attacks?”arXiv preprint arXiv:2309.11751, 2023

    2023 arXiv

  41. [49]

    Re- thinking model ensemble in transfer-based adversarial attacks,

    H. Chen, Y. Zhang, Y. Dong, X. Yang, H. Su, and J. Zhu, “Re- thinking model ensemble in transfer-based adversarial attacks,” inICLR, 2024

    2024

  42. [50]

    Physical adversarial attack meets computer vision: A decade survey,

    H. Wei, H. Tang, X. Jia, Z. Wang, H. Yu, Z. Li, S. Satoh, L. Van Gool, and Z. Wang, “Physical adversarial attack meets computer vision: A decade survey,”IEEE TP AMI, 2024

    2024

  43. [51]

    Robust adversarial objects against deep learning models,

    T. Tsai, K. Yang, T.-Y. Ho, and Y. Jin, “Robust adversarial objects against deep learning models,” inAAAI, vol. 34, no. 01, 2020, pp. 954–962

    2020

  44. [52]

    Pointcloud saliency maps,

    T. Zheng, C. Chen, J. Yuan, B. Li, and K. Ren, “Pointcloud saliency maps,” inCVPR, 2019, pp. 1598–1606

    2019

  45. [53]

    Advpc: Transfer- able adversarial perturbations on 3d point clouds,

    A. Hamdi, S. Rojas, A. Thabet, and B. Ghanem, “Advpc: Transfer- able adversarial perturbations on 3d point clouds,” inECCV, 2020, pp. 241–257

    2020

  46. [54]

    Lg-gan: Label guided adversarial network for flexible targeted attack of point cloud based deep networks,

    H. Zhou, D. Chen, J. Liao, K. Chen, X. Dong, K. Liu, W. Zhang, G. Hua, and N. Yu, “Lg-gan: Label guided adversarial network for flexible targeted attack of point cloud based deep networks,” inCVPR, 2020, pp. 10 356–10 365

    2020

  47. [55]

    Dta: Physical camouflage attacks using differentiable transformation network,

    N. Suryanto, Y. Kim, H. Kang, H. T. Larasati, Y. Yun, T.-T.-H. Le, H. Yang, S.-Y. Oh, and H. Kim, “Dta: Physical camouflage attacks using differentiable transformation network,” inCVPR, 2022, pp. 15 305–15 314

    2022

  48. [56]

    Active: Towards highly trans- ferable 3d physical camouflage for universal and robust vehicle evasion,

    N. Suryanto, Y. Kim, H. T. Larasati, H. Kang, T.-T.-H. Le, Y. Hong, H. Yang, S.-Y. Oh, and H. Kim, “Active: Towards highly trans- ferable 3d physical camouflage for universal and robust vehicle evasion,” inCVPR, 2023, pp. 4305–4314

    2023

  49. [57]

    Learning from simulated and unsupervised images through adversarial training,

    A. Shrivastava, T. Pfister, O. Tuzel, J. Susskind, W. Wang, and R. Webb, “Learning from simulated and unsupervised images through adversarial training,” inCVPR, 2017, pp. 2107–2116

    2017

  50. [58]

    Pointnet: Deep learning on point sets for 3d classification and segmentation,

    C. R. Qi, H. Su, K. Mo, and L. J. Guibas, “Pointnet: Deep learning on point sets for 3d classification and segmentation,” inCVPR, 2017, pp. 652–660

    2017

  51. [59]

    Pointnet++: Deep hierarchical feature learning on point sets in a metric space,

    C. R. Qi, L. Yi, H. Su, and L. J. Guibas, “Pointnet++: Deep hierarchical feature learning on point sets in a metric space,” in NeurIPS, I. Guyon, U. V . Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds., vol. 30. Curran Associates, Inc., 2017

    2017

  52. [60]

    Faster r-cnn: Towards real-time object detection with region proposal networks,

    S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,”IEEE TPMAI, vol. 39, no. 6, pp. 1137–1149, 2016

    2016

  53. [61]

    You only look once: Unified, real-time object detec- tion,

    J. Redmon, “You only look once: Unified, real-time object detec- tion,” inCVPR, 2016

    2016

  54. [62]

    Beyond the nav-graph: Vision-and-language navigation in continuous en- vironments,

    J. Krantz, E. Wijmans, A. Majumdar, D. Batra, and S. Lee, “Beyond the nav-graph: Vision-and-language navigation in continuous en- vironments,” inECCV, 2020, pp. 104–120

    2020

  55. [63]

    A frontier-based approach for autonomous explo- ration,

    B. Yamauchi, “A frontier-based approach for autonomous explo- ration,” inCIRA. IEEE, 1997, pp. 146–151

    1997

  56. [64]

    Large language models can be easily distracted by irrelevant context,

    F. Shi, X. Chen, K. Misra, N. Scales, D. Dohan, E. H. Chi, N. Sch¨arli, and D. Zhou, “Large language models can be easily distracted by irrelevant context,” inICML. PMLR, 2023, pp. 31 210–31 227

    2023

  57. [65]

    Ad- versarial camouflage: Hiding physical-world attacks with natural styles,

    R. Duan, X. Ma, Y. Wang, J. Bailey, A. K. Qin, and Y. Yang, “Ad- versarial camouflage: Hiding physical-world attacks with natural styles,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 1000–1008

    2020

  58. [66]

    An overview on 3d printing technology: Technological, materials, and applications,

    N. Shahrubudin, T. C. Lee, and R. Ramlan, “An overview on 3d printing technology: Technological, materials, and applications,” Procedia manufacturing, vol. 35, pp. 1286–1296, 2019

    2019

  59. [67]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection,

    S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, C. Li, J. Yang, H. Su, J. Zhuet al., “Grounding dino: Marrying dino with grounded pre-training for open-set object detection,”arXiv preprint arXiv:2303.05499, 2023

    2023 arXiv

  60. [68]

    A density-based algorithm for discovering clusters in large spatial databases with noise,

    M. Ester, H.-P . Kriegel, J. Sander, X. Xuet al., “A density-based algorithm for discovering clusters in large spatial databases with noise,” inKDD, vol. 96, no. 34, 1996, pp. 226–231

    1996

  61. [69]

    Matterport3d: Learning from rgb-d data in indoor environments,

    A. Chang, A. Dai, T. Funkhouser, M. Halber, M. Niebner, M. Savva, S. Song, A. Zeng, and Y. Zhang, “Matterport3d: Learning from rgb-d data in indoor environments,” in3DV, 2017, pp. 667–676

    2017

  62. [70]

    Language-driven semantic segmentation,

    B. Li, K. Q. Weinberger, S. Belongie, V . Koltun, and R. Ranftl, “Language-driven semantic segmentation,” inICLR, 2022

    2022

  63. [71]

    Habitat-matterport 3d dataset (HM3d): 1000 large-scale 3d environments for embod- ied AI,

    S. K. Ramakrishnan, A. Gokaslan, E. Wijmans, O. Maksymets, A. Clegg, J. M. Turner, E. Undersander, W. Galuba, A. Westbury, A. X. Chang, M. Savva, Y. Zhao, and D. Batra, “Habitat-matterport 3d dataset (HM3d): 1000 large-scale 3d environments for embod- ied AI,” inNeurIPS, 2021

    2021

  64. [72]

    Habitat: A platform for embodied ai research,

    M. Savva, A. Kadian, O. Maksymets, Y. Zhao, E. Wijmans, B. Jain, J. Straub, J. Liu, V . Koltun, J. Maliket al., “Habitat: A platform for embodied ai research,” inProceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 9339–9347

    2019

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.