Vision Token Manipulation Attacks on Cloud-Edge Inference of Large Vision-Language Models

Jiahao Xu; Olivera Kotevska; Rui Hu; Zikai Zhang

REVIEW 2 major objections 4 minor 18 references

Vision Token Manipulation Attacks on Cloud-Edge Inference of Large Vision-Language Models

T0 review · 2 major / 4 minor · reviewed 2026-07-12 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Manipulating only 10% of vision tokens in cloud-edge LVLM inference can cut accuracy by up to 88%.

desk verdict Budgeted MITM on vision tokens is a real, well-demonstrated attack surface for cloud-edge LVLMs; the attention selector is a useful heuristic whose advantage is real but modest and under-validated. read the letter →

arxiv 2607.02819 v1 pith:ARDSGIDQ submitted 2026-07-02 cs.CR cs.AIcs.CV

Zikai Zhang , Rui Hu , Olivera Kotevska , Jiahao Xu This is my paper · ORCID

classification cs.CRcs.AIcs.CV

keywords cloud-edgeinferencelargevision-languagemodelsvisiontokenmanipulationman-in-the-middleattackself-attentiondisruptionsecureedgeintelligence

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Cloud-edge systems for large vision-language models send compact vision tokens from a light edge encoder to a heavy cloud language model. The paper shows that a black-box man-in-the-middle adversary who intercepts that stream can flip, mask, noise, or reorder a small budget of those tokens and drive accuracy near zero. Four simple manipulations already cause large drops; an optimization that picks tokens to maximize self-attention disruption while avoiding high-norm bias makes the attack still stronger. Across six models (3B–72B) and four standard benchmarks the worst-case drop reaches 88 percentage points when only one-tenth of the tokens are touched, and the attack itself costs only milliseconds. The work therefore establishes that the vision-token link is a practical, high-leverage attack surface that current cloud-edge designs leave unprotected.

What carries the argument

VTM-Attack together with the continuous relaxation of token selection that maximises the Frobenius gap between original and manipulated self-attention matrices while subtracting a norm-aware penalty (Eqs. 2–4 and Algorithm 1).

What would settle it

Re-run the same sign-flip and optimisation-based attacks on an LVLM whose intermediate vision tokens are protected by authenticated encryption or whose vision encoder is fused with the language backbone so that no free-standing token stream exists; if accuracy remains near the benign baseline, the claimed vulnerability disappears.

Watch

Extended reading notes

Core claim

Under a black-box man-in-the-middle model that may alter at most a fraction ρ of the vision tokens while preserving their shape, simple value-based manipulations—especially sign flip—already collapse accuracy on multiple LVLMs; an optimization that selects the subset maximizing self-attention change under a norm-aware regularizer further amplifies the damage, showing that 10 % token manipulation is sufficient to reduce accuracy by as much as 88.31 %.

Load-bearing premise

Self-attention computed only on the intercepted vision-token matrix is a faithful enough black-box proxy for how those tokens later affect the full model’s output distribution.

Editorial extensions

If this is right

Any cloud-edge LVLM pipeline that transmits unprotected vision tokens is immediately open to high-impact, low-latency sabotage.
Defences must either encrypt or authenticate the token stream, or redesign the split so that no semantically rich intermediate representation leaves the edge.
Model families differ sharply in robustness (Qwen collapses far more readily than InternVL), so architecture choice itself becomes a security decision.
Attack cost is negligible compared with 4G/5G transmission latency, so the adversary can act without introducing detectable delay.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same self-attention proxy could be inverted to produce a cheap token-level integrity check that flags anomalous attention patterns before decoding begins.
Because the attack works without model weights, similar token-stream vulnerabilities may appear in any split-inference system that ships intermediate embeddings, not only vision-language models.
Sign-flip’s extreme effectiveness suggests that directional (rather than magnitude) information is the dominant carrier of visual semantics inside current LVLMs.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The punchline is simple: once you split an LVLM so that vision tokens leave the edge, a black-box MITM that flips the sign of only 10 % of them can drive accuracy from the high 80s to near zero on several Qwen models. That is a concrete, high-impact finding for a deployment pattern that is already being productized.

What is new is the attack surface itself. Prior work hits raw images or intermediate text; here the adversary never sees pixels or model weights and still collapses performance under a strict token budget. The four naïve operators (permutation, masking, Gaussian, sign-flip) are cleanly defined, and the multi-model, multi-benchmark tables (six LVLMs, four tasks, full ρ sweeps in the appendix) make the vulnerability hard to dismiss. Sign-flip is consistently the most destructive, which matches the geometry they describe. The wall-clock comparison showing that even the optimized selector finishes in tens of milliseconds is a nice practical touch.

The soft spots are real but secondary. The optimization objective maximizes self-attention disruption computed only on the vision Gram matrix; no text tokens, cross-attention, or output loss ever enter. Appendix C notes the high-norm bias and LayerNorm, yet they never check whether the Top-k set actually coincides with the tokens that most change P(y|·). Because naïve sign-flip already destroys several models, the vulnerability claim stands without the selector. The selector’s reported gains are therefore modest improvements over random, not the load-bearing novelty. Missing multi-seed error bars on the random baselines and unreleased attack code are ordinary reproducibility gaps, not fatal ones. Mitigations are mentioned only in the conclusion.

This is for people building or securing split multimodal systems. The math is elementary, the data are broad, and the citation pattern is fair. I would send it to peer review; a referee can demand the correlation check and code release without questioning the core empirical result. Worth reading and, once the artifacts appear, worth citing when you discuss edge-AI threat models.

Referee Report

2 major / 4 minor

Summary. The paper identifies the transmission of intermediate vision tokens from edge to cloud in partitioned LVLM inference as a new man-in-the-middle attack surface. Under a black-box budget constraint (token manipulation ratio ρ), it defines four naïve VTM-Attacks (permutation, masking, Gaussian perturbation, sign flip) that act on a random subset of tokens, plus an optimization-based selector that maximizes a self-attention disruption loss L_SA regularized by a norm-aware term L_norm (Eqs. 2–4) via projected gradient ascent on continuous weights w, followed by Top-k. Experiments on six Qwen2.5-VL and InternVL3.5 models (3B–72B) across OCRBench, MMBench, HallusionBench and MathVista show that ρ=0.1 already produces large accuracy collapses (up to 88.31 percentage points under optimized sign flip), with ablations on ρ, objective components and wall-clock cost.

Significance. If the reported effect sizes hold, the work establishes a practically relevant vulnerability for an increasingly common deployment pattern (edge vision encoder + cloud LLM). The evaluation is unusually broad for an attack paper—six models spanning two families, four diverse benchmarks, systematic ρ sweeps, objective ablations and latency comparisons—and the quantitative drops are large enough to be actionable for system designers. The optimization procedure, while heuristic, is a concrete black-box contribution that measurably improves over random selection on several model–benchmark pairs. The manuscript therefore supplies both a clear threat model and reproducible empirical evidence that defenses at the vision-token interface are needed.

major comments (2)

[Section III-C, Eqs. (2)–(4)] Section III-C and Eqs. (2)–(4) (with continuous relaxation in Appendix B): L_SA is defined solely on the vision-token Gram matrix Attn(V)=softmax(VV⊤/√d)V; no text tokens, cross-attention or downstream LLM layers enter the objective. While Table I and Fig. 3 show that the resulting Top-k sets improve attack success over random selection, the paper never directly checks whether those indices coincide with tokens of highest influence on P(y_t|V,P,Y_<t;θ). A simple white-box or leave-one-out correlation study (even on a single model) would confirm that the proxy is not merely selecting high-norm or high-variance tokens that happen to work under sign flip.
[Table I] Table I (and the corresponding ρ-sweep tables in Appendix A): All naïve-attack numbers appear to be single-run point estimates with no standard deviation or multi-seed statistics. Because the optimization gains are sometimes modest (a few points) and the random baseline itself is stochastic, it is impossible to judge whether the reported improvements are statistically reliable. Reporting mean±std over at least 3–5 independent index draws for the random columns would make the claim that the selector “significantly improves attack effectiveness” quantitatively defensible.

minor comments (4)

[Fig. 2] Fig. 2 and the Appendix A tables: the y-axis scales and line styles make it hard to compare absolute accuracy across model families at a glance; a common y-range or an additional relative-drop panel would improve readability.
[Section II] Notation: the composite signal is introduced as T:={V,S} but later the cloud receives T and tokenizes S; a short clarifying sentence that S already contains the visual placeholders would avoid momentary confusion.
[Section IV-D] Several OCR/encoding artifacts remain (e.g., “naïve” rendered with diaeresis, “TMA-Attacks” vs. “VTM-Attacks” in the efficiency subsection). A final proof-reading pass is needed.
[Appendix D] Appendix D sets α=0.5 without sensitivity analysis; a one-sentence note on how results change for α∈{0.25,1.0} would reassure readers that the adaptive-λ scheme is not brittle.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: attack effectiveness is measured on external benchmarks; the self-attention proxy is an independent heuristic whose success is judged empirically, not by construction.

full rationale

The paper's central quantitative claims (accuracy collapses of up to 88.31% under 10% token manipulation) are obtained by running the proposed attacks on public benchmarks (OCRBench, MMBench, HallusionBench, MathVista) with six off-the-shelf LVLMs and reporting end-task accuracy. The optimization-based selector (Eqs. 2–4, Algorithm 1) defines a continuous proxy L_SA(w) − λ L_norm(w) solely on the intercepted vision-token matrix V; this proxy is never fitted to the accuracy numbers that are later reported, nor is any free parameter tuned so that the claimed drop becomes true by definition. λ is set by a single adaptive gradient-magnitude ratio (Appendix D) that does not involve the evaluation metrics. The continuous relaxation of sign-flip (Appendix B) and the norm-bias analysis (Appendix C) are internal to the selection procedure and do not re-derive or presuppose the measured accuracy drops. There are no load-bearing self-citations of uniqueness theorems, no ansatz smuggled from prior author work that forces the result, and no renaming of a known empirical pattern presented as a new derivation. The entire evaluation chain is therefore self-contained against external data; any weakness of the self-attention proxy is a correctness/validity concern, not circularity.

Assumptions & free parameters 4 free parameters · 3 assumptions · 2 invented entities

The central claim is an empirical security result that rests on a standard MITM threat model, public model architectures, and a handful of hand-chosen optimization hyper-parameters; no new physical or mathematical entities beyond the attack constructions themselves are required.

free parameters (4)

token manipulation ratio ρ = 0.1 (default)
Fraction of vision tokens the attacker is allowed to change; main tables fix ρ=0.1 and ablate 0.1–1.0.
optimization iterations T = 30
Number of projected-gradient steps used to obtain the continuous selection weights w.
learning rate η = 0.05
Step size for updating the continuous selection vector w.
regularization scale α = 0.5
Multiplier that sets adaptive λ = α · (mean |∇L_SA| / mean |∇L_norm|) so the two loss terms start with comparable gradient magnitudes.

assumptions (3)

domain assumption An adversary can intercept the composite edge-to-cloud signal T={V,S} and replace V by a dimension-preserving ˆV without immediate detection, remaining black-box with respect to both edge and cloud model parameters.
Stated as the attack model in Section III-A; the entire experimental campaign is conducted under this premise.
ad hoc to paper Self-attention computed only on the vision-token matrix is a useful surrogate for the tokens’ influence on the full LVLM output distribution.
Justifies the L_SA objective (Eq. 2) that drives the optimization-based selector; validated only empirically inside the paper.
domain assumption Preserving the exact shape L_v × d of the vision-token matrix is necessary and sufficient for the cloud pipeline to accept the tokens without raising an anomaly.
Constraint (i) of the attacker’s capability list in Section III-A.

invented entities (2)

VTM-Attack family (permutation / masking / Gaussian / sign-flip + attention-based selection)
purpose: Concrete realizations of the budgeted MITM threat model
Defined and evaluated only inside this paper; no external falsifiable prediction is offered beyond the reported accuracy drops.
self-attention disruption loss L_SA regularized by norm-aware term L_norm
purpose: Black-box proxy objective for ranking which tokens to manipulate
Introduced in Eqs. (2)–(4); its utility is measured solely by the paper’s own end-task accuracy numbers.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Vision Token Manipulation Attacks on Cloud-Edge Inference of Large Vision-Language Models." pith.science (2026). https://pith.science/paper/ARDSGIDQ

@misc{pith2026260702819,
  author       = {Pith},
  title        = {Pith review of: Vision Token Manipulation Attacks on Cloud-Edge Inference of Large Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ARDSGIDQ}},
  note         = {Machine review of arXiv:2607.02819}
}

read the original abstract

Cloud-edge Large Vision-Language Model (LVLM) inference enables efficient deployment by splitting computation between edge devices and cloud servers. In this process, intermediate vision tokens are transmitted from the edge to the cloud over a communication link, thereby exposing a new attack surface. We study vision token manipulation attack (VTM-Attack) under a black-box man-in-the-middle setting, where an adversary intercepts and manipulates a subset of transmitted vision tokens under a budget constraint. We propose four na\"ive attack strategies and an optimization-based token selection method. Experiments on 6 state-of-the-art LVLMs (3B-72B) across 4 benchmarks show that manipulating only 10\% of vision tokens can reduce accuracy by up to 88.31\%. These results reveal a critical vulnerability in cloud-edge LVLM inference.

Figures

Figures reproduced from arXiv: 2607.02819 by the authors.

**Figure 2.** Performance of na¨ıve VTM-Attacks under varying TMR on OCRBench. high-performance computing cluster with 20 nodes in parallel, each equipped with four AMD MI250X GPUs (128 GB). B. Main Results In Table I, we report the main experimental results. Overall, the benign performance across all benchmarks is consistently high, yet manipulating vision tokens leads to severe degradation in LVLM performance. Na¨ıve VTM-Attac… view at source ↗

**Figure 3.** Effectiveness of optimization objectives on OCRBench. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Comparison of communication latency and VTM-Attacks time cost. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 5 linked inside Pith

[1]

Qwen2.5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2.5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

arXiv 2025
[2]

A survey of man in the middle attacks.IEEE communications surveys & tutorials, 2016

Mauro Conti, Nicola Dragoni, and Viktor Lesyk. A survey of man in the middle attacks.IEEE communications surveys & tutorials, 2016

2016
[3]

Vlmevalkit: An open-source toolkit for evaluating large multi- modality models

Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, et al. Vlmevalkit: An open-source toolkit for evaluating large multi- modality models. InProceedings of the 32nd ACM International Conference on Multimedia, pages 11198–11201, 2024

2024
[4]

On the robustness of split learning against adversarial attacks

Mingyuan Fan, Cen Chen, Chengyu Wang, Wenmeng Zhou, and Jun Huang. On the robustness of split learning against adversarial attacks. arXiv preprint arXiv:2307.07916, 2023

arXiv 2023
[5]

Explaining and harnessing adversarial examples.arXiv:1412.6572, 2014

Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples.arXiv:1412.6572, 2014

arXiv 2014
[6]

Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models

Tianrui Guan, Fuxiao Liu, et al. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14375–14385, 2024

2024
[7]

Hyperion: Low-latency ultra-hd video analytics via collaborative vision transformer inference

Linyi Jiang, Yifei Zhu, Hao Yin, and Bo Li. Hyperion: Low-latency ultra-hd video analytics via collaborative vision transformer inference. arXiv preprint arXiv:2512.21730, 2025

arXiv 2025
[8]

Backdoorvlm: A benchmark for backdoor attacks on vision-language models.arXiv preprint arXiv:2511.18921, 2025

Juncheng Li, Yige Li, Hanxun Huang, Yunhao Chen, Xin Wang, et al. Backdoorvlm: A benchmark for backdoor attacks on vision-language models.arXiv preprint arXiv:2511.18921, 2025

arXiv 2025

Show all 18 references

[9]

Distributed vlms: Efficient vision-language processing through cloud-edge collabo- ration

Yuyang Li, Devika Gumaste, Mehmet Kerem Turkcan, et al. Distributed vlms: Efficient vision-language processing through cloud-edge collabo- ration. In2025 IEEE International Conference on Pervasive Computing and Communications Workshops, pages 280–286, 2025

2025
[10]

Mmbench: Is your multi-modal model an all-around player? InEuropean conference on computer vision, pages 216–233

Yuan Liu, Haodong Duan, Yuanhan Zhang, et al. Mmbench: Is your multi-modal model an all-around player? InEuropean conference on computer vision, pages 216–233. Springer, 2024

2024
[11]

Ocrbench: on the hidden mystery of ocr in large multimodal models.Science China Information Sciences, 67(12):220102, 2024

Yuliang Liu, Zhang Li, Mingxin Huang, et al. Ocrbench: on the hidden mystery of ocr in large multimodal models.Science China Information Sciences, 67(12):220102, 2024

2024
[12]

Formalizing and benchmarking prompt injection attacks and defenses

Yupei Liu, Yuqi Jia, Runpeng Geng, Jinyuan Jia, and Neil Zhenqiang Gong. Formalizing and benchmarking prompt injection attacks and defenses. In33rd USENIX Security Symposium, pages 1831–1847, 2024

2024
[13]

Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts

Pan Lu, Hritik Bansal, Tony Xia, et al. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv:2310.02255, 2023

2023 arXiv
[14]

Prompt inference attack on distributed large language model inference frameworks

Xinjian Luo, Ting Yu, and Xiaokui Xiao. Prompt inference attack on distributed large language model inference frameworks. InProceedings of the 2025 ACM SIGSAC Conference on Computer and Communica- tions Security, pages 1739–1753, 2025

2025
[15]

Net- gpt: A llm-empowered man-in-the-middle chatbot for unmanned aerial vehicle

Brett Piggott, Siddhant Patil, Guohuan Feng, Ibrahim Odat, et al. Net- gpt: A llm-empowered man-in-the-middle chatbot for unmanned aerial vehicle. InProceedings of the Eighth ACM/IEEE Symposium on Edge Computing, pages 287–293, 2023

2023
[16]

edgevlm: Cloud-edge collaborative real-time vlm based on context transfer.arXiv preprint arXiv:2508.12638, 2025

Chen Qian, Xinran Yu, Zewen Huang, Danyang Li, et al. edgevlm: Cloud-edge collaborative real-time vlm based on context transfer.arXiv preprint arXiv:2508.12638, 2025

2025
[17]

Internvl3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency.arXiv preprint arXiv:2508.18265, 2025

Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, et al. Internvl3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency.arXiv preprint arXiv:2508.18265, 2025

2025 arXiv
[18]

vision token manipulation attacks on cloud-edge inference of large vision-language models

Zikai Zhang et al. Appendix for “vision token manipulation attacks on cloud-edge inference of large vision-language models”. https://github.com/superkevingit/Vision-Token-Manipulation-Attack/ blob/main/appendix.pdf, 2026. APPENDIX This appendix provides additional experimental...

2026

Pith tools

Reviewed July 12, 2026 · model on record in the stance chip above.

[1] [1]

Qwen2.5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2.5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

arXiv 2025

[2] [2]

A survey of man in the middle attacks.IEEE communications surveys & tutorials, 2016

Mauro Conti, Nicola Dragoni, and Viktor Lesyk. A survey of man in the middle attacks.IEEE communications surveys & tutorials, 2016

2016

[3] [3]

Vlmevalkit: An open-source toolkit for evaluating large multi- modality models

Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, et al. Vlmevalkit: An open-source toolkit for evaluating large multi- modality models. InProceedings of the 32nd ACM International Conference on Multimedia, pages 11198–11201, 2024

2024

[4] [4]

On the robustness of split learning against adversarial attacks

Mingyuan Fan, Cen Chen, Chengyu Wang, Wenmeng Zhou, and Jun Huang. On the robustness of split learning against adversarial attacks. arXiv preprint arXiv:2307.07916, 2023

arXiv 2023

[5] [5]

Explaining and harnessing adversarial examples.arXiv:1412.6572, 2014

Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples.arXiv:1412.6572, 2014

arXiv 2014

[6] [6]

Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models

Tianrui Guan, Fuxiao Liu, et al. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14375–14385, 2024

2024

[7] [7]

Hyperion: Low-latency ultra-hd video analytics via collaborative vision transformer inference

Linyi Jiang, Yifei Zhu, Hao Yin, and Bo Li. Hyperion: Low-latency ultra-hd video analytics via collaborative vision transformer inference. arXiv preprint arXiv:2512.21730, 2025

arXiv 2025

[8] [8]

Backdoorvlm: A benchmark for backdoor attacks on vision-language models.arXiv preprint arXiv:2511.18921, 2025

Juncheng Li, Yige Li, Hanxun Huang, Yunhao Chen, Xin Wang, et al. Backdoorvlm: A benchmark for backdoor attacks on vision-language models.arXiv preprint arXiv:2511.18921, 2025

arXiv 2025