REVIEW 5 major objections 5 minor 37 references
AGMark: Attention-Guided Dynamic Watermarking for Large Vision-Language Models
T0 review · 5 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash
Pith's one-line read Watermarking vision-language output can get sharper by protecting the tokens the model actually attends to.
desk verdict A well-engineered dynamic LVLM watermark whose central detectability claim is currently unverifiable because the detection algorithm is never defined. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The core mechanism is the semantic critical weight ψ_t(k), computed per decoding step t and vocabulary token k. It is a convex combination (weighted by ω) of two z-scored cosine similarities: a vision critical weight — the cosine similarity between the attention-weighted average of visual embeddings (weights from the last-layer attention over image tokens) and the token embedding — and a context critical weight — the cosine similarity between the current hidden state and the token embedding. After min-max normalization and sorting, a subset of top tokens is designated 'semantic critical'. The adaptive vocabulary partitioner then sets the protected ratio η_t = α · ρ_t · (1 − H^norm_t), where
What would settle it
A controlled experiment that replaces the attention-guided critical weights with random or fixed-ranking token selection (keeping the same adaptive ratio and entropy terms) and finds the same CHAIR/BLEU improvement would falsify the claim that attention-guided dynamic selection is the cause of the fidelity gains.
Extended reading notes
Core claim
On its own terms, the paper claims that dynamic, attention-guided selection of 'semantic critical tokens' at each decoding step outperforms both vision-agnostic watermarking (which biases all tokens equally) and recent vision-specific watermarking that computes criticality once (static). The central result is that AGMark improves perplexity, BLEU, semantic alignment, and hallucination (CHAIR) across three 8B LVLMs on AMBER and MS-COCO, while maintaining detection AUC ≥99.36% and attack AUC ≥88.61% under paraphrase. The paper presents ablations showing that removing attention-guidance, vision weight, context weight, entropy, density, or the adaptive scale each degrades quality, and argues tha
Load-bearing premise
The method stands or falls on the assumption that the cosine-similarity semantic critical weight (equations 3–6) is a reliable proxy for which tokens must be protected to preserve visual semantics; the paper does not validate this proxy against ground-truth token criticality.
Editorial extensions
If this is right
- Watermark detectors for LVLMs can be built around dynamic visual evidence rather than fixed vocabulary partitions, allowing detectability to survive paraphrase attacks better than static vision-specific methods.
- Generation quality and hallucination metrics can improve as a side effect of watermarking, because the bias pushes sampling toward tokens grounded in the attended image regions.
- The same adaptive swapping framework could be applied to text-only LLMs by replacing vision critical weights with other evidence signals (e.g., retrieved context or user prompt tokens).
- Long-generation scenarios (100-200 tokens) may see particularly large fidelity gains, since the method re-computes criticality at each step instead of relying on a stale initial estimate.
- The ratio formula η_t = α·ρ_t·(1−H) suggests a general principle: watermark injection strength should be throttled by both model uncertainty and how concentrated the evidence distribution is.
Reading between the lines
- The criticality proxy is unvalidated against ground truth; a natural extension would test whether the protected tokens actually correspond to salient image regions (e.g., via grounding annotations or human ratings), which would separate the attention signal's value from the adaptive ratio's value.
- Because the method uses only cosine similarities and attention weights, it may transfer to other multimodal settings (e.g., video generation, audio captioning) or to instruction-following LLMs with retrieval context, but the paper does not test those settings.
- The reported CHAIR gains might partly stem from the entropy term (low-entropy steps favor accurate tokens), independent of the vision attention; an ablation with only context weights and entropy would isolate whether visual attention is truly load-bearing.
- If the method is applied at scale, the per-step matrix operations and sorting may become a bottleneck; the paper reports modest latency increases, but a careful cost-benefit analysis for interactive deployment is left open.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript proposes AGMark, a logits-based watermarking framework for large vision-language models (LVLMs). At each decoding step, AGMark computes per-token semantic-critical weights by fusing vision attention-weighted similarities (Eq. 3) with hidden-state/token similarities (Eq. 4), then adaptively swaps high-weight tokens into the green list, with the protected-token ratio determined by token entropy and weight density (Eqs. 9–15). Experiments on three 8B LVLMs and two benchmarks compare against KGW, SynthID, IE, MorphMark, and VLA-Mark, reporting improved PPL/BLEU/BertScore/CHAIR while maintaining AUC ≥ 99.36%, plus robustness to five text-space attacks. The paper includes ablations, hyperparameter analysis, and latency measurements.
Significance. If the reported gains hold, AGMark addresses a real problem: vision-agnostic watermarks can degrade LVLM visual grounding and text quality. The dynamic, attention-guided selection of protected tokens is well motivated, and the evaluation spans multiple models, datasets, and baselines. The paper also ships useful ablations and a detailed efficiency comparison. However, the manuscript currently omits the detection algorithm, which is load-bearing for the detectability claims, and the hyperparameters are tuned on the same benchmark used for the headline results. These issues are fixable and the core idea is publishable after major revision.
major comments (5)
- [§3.2, Eq. (15); §5.1, Table 1; Fig. 5] The paper defines the green/red partition as a function of model-internal quantities — A^v_t, E_h, and the sorted ψ̃_t — but never specifies the detection statistic. A detector must reconstruct the exact dynamic green list for each position to compute a green-token count or z-score. This requires white-box access to the watermarked LVLM and exact knowledge of all hyperparameters (ω, α, τ, δ, γ, plus the optional margin/cap). The manuscript does not state these assumptions, give detection pseudocode, or discuss the black-box setting. Without this, the AUC/ACC numbers in Table 1 and Fig. 5 are not independently reproducible.
- [§5.4, Table 2 vs. §5.1, Table 1] The optimal hyperparameters (ω=0.50, α=0.27, τ=0.98) are selected by sweeping on Llava-Next/AMBER, and the same model/dataset is then used in Table 1 to demonstrate the headline quality gains. This is selection on the evaluation set, so the reported improvements over baselines may partly reflect a tuning advantage. Please use a separate validation split for hyperparameter selection, or report results with fixed hyperparameters across all models and datasets, and ensure all baselines receive the same tuning protocol.
- [§5.1, Table 1] No variance or significance testing is reported. The headline improvements are small in some metrics (e.g., PPL 4.80 vs. 4.98, CHAIR 6.2% vs. 7.9%), and all results use a single seed. Without confidence intervals or multiple seeds, it is difficult to assess whether the differences are stable rather than noise. Please report variance across seeds or bootstrap intervals, and where possible paired significance tests.
- [§3.1, Eqs. (3)–(6); Table 3] The central mechanism rests on a heuristic definition of semantic-critical tokens as those with high cosine similarity in the vision-attention-weighted embedding space and to the current hidden state. The ablations show that removing these components hurts AGMark's own metrics, but this does not externally validate the proxy; it only shows that the components matter within this design. Please provide external validation (e.g., correlation with ground-truth caption entities or human importance judgments) or compare against a random-token baseline with the same protected-token budget. Without this, the causal role of the attention-guided selection in visual-fidelity improvement is not fully established.
- [§3.2, Eq. (15)] The text mentions 'optionally gating the swap by a margin threshold and a per-step cap to avoid oscillation,' but this gating is never defined, and no ablation or hyperparameter values are given. If it is used in the experiments, the exact rule and values must be specified; if it is not used, the clause should be removed. As written, it prevents exact reproduction of the method.
minor comments (5)
- [§6.1] Typo: 'MorkphMark' should be 'MorphMark'.
- [Fig. 4 caption] Typo: 'Mddle' should be 'Middle' in the BertScore panel label.
- [§3.2] Typo: 'multi-model input' should be 'multimodal input'.
- [§3.1, Eq. (3)] The notation A^v_t^T · E_v is ambiguous without specifying shapes. Please clarify whether A^v_t is a vector over vision tokens or a matrix, and define the multiplication explicitly.
- [Table 3] The deltas (e.g., '+2.30' after '8.50') are not defined in the caption. Please clarify that they represent the difference relative to the Full model, or restructure the table to make this explicit.
Circularity Check
No significant circularity: AGMark is an empirical heuristic pipeline whose claims are measured outcomes, not by-construction consequences of its definitions.
full rationale
AGMark is an empirical watermarking pipeline. Equations (3)-(6) define a heuristic semantic-critical weight, Eqs. (11)-(15) define an adaptive green/red-list partition, and Eq. (16) is the standard KGW-style biased sampling. No claimed result is derived from those definitions by construction: the reported quality, fidelity, detectability, and robustness numbers are measured outcomes, not closed-form consequences of the fitted hyperparameters. The paper's self-citations ([16], [32], [33]) are background references and are not load-bearing: [32] is cited only for using the last-layer hidden state, [16] for motivation, and [33] for attack context; there is no uniqueness theorem or ansatz imported from prior self-cited work that forces the design. The absence of a fully specified detection statistic is a reproducibility gap, but it is not circularity: no equation in the paper reduces to its own input, and the central mechanism is not defined in terms of the evaluation targets. The hyperparameter selection on the same benchmark (Table 2) is a validation-protocol weakness and a source of selection bias, but it is not a by-construction equivalence; the cross-model and cross-dataset rows (MS-COCO, Qwen3-VL, InternVL-3.5) provide independent evidence for the main qualitative claims. Therefore no significant circularity is found.
Assumptions & free parameters
free parameters (4)
- ω (fusion weight) =
0.50
- α (base critical-token ratio) =
0.27
- τ (density threshold) =
0.98
- δ (watermark bias) =
tuned per method to achieve AUC ≥ 99%
assumptions (4)
- domain assumption Attention weights over visual regions reflect the visual information relevant for generating the current token (used in Eq. 3 to define vision critical weights).
- domain assumption Cosine similarity between attended vision embeddings and vocabulary token embeddings in the shared embedding space is a valid measure of a token's visual relevance (Eq. 3).
- domain assumption The hidden-state/token-embedding cosine similarity (Eq. 4) captures contextual coherence needed for the current generation step.
- domain assumption The standard KGW-style detection statistic remains valid when green/red lists are adaptively swapped per step using model states.
Cite this review
Pith. "Pith review of AGMark: Attention-Guided Dynamic Watermarking for Large Vision-Language Models." pith.science (2026). https://pith.science/paper/AIX6UKHB
@misc{pith2026260209611,
author = {Pith},
title = {Pith review of: AGMark: Attention-Guided Dynamic Watermarking for Large Vision-Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/AIX6UKHB}},
note = {Machine review of arXiv:2602.09611}
}
read the original abstract
Watermarking has emerged as a pivotal solution for content traceability and intellectual property protection in large vision language models (LVLMs). However, vision-agnostic watermarks may introduce visually irrelevant tokens and disrupt visual grounding by enforcing indiscriminate pseudo-random biases. Additionally, current vision-specific watermarks rely on a static, one-time estimation of vision-critical weights and ignore the weight distribution density when determining the proportion of protected tokens. This design fails to account for dynamic changes in visual dependence during generation and may introduce low-quality tokens in the long tail. To address these challenges, we propose Attention-Guided Dynamic Watermarking (AGMark), a novel framework that embeds detectable signals while largely preserving visual-semantic fidelity. At each decoding step, AGMark first dynamically identifies semantic-critical evidence based on attention weights for visual relevance, together with context-aware coherence cues, resulting in a more adaptive and well-calibrated evidence-weight distribution. It then determines the proportion of semantic-critical tokens by jointly considering uncertainty awareness (token entropy) and evidence calibration (weight density), thereby enabling more reliable adaptive vocabulary partitioning to avoid irrelevant tokens. Empirical results consistently confirm that AGMark outperforms conventional methods, substantially improving generation quality and yielding particularly strong gains in visual semantic fidelity in the later stages of generation. Our framework maintains highly competitive detection performance (at least 99.36% AUC) and robust attack resilience (at least 88.61% AUC) without sacrificing inference efficiency, taking a significant step toward reliability-preserving multimodal watermarking.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018. Bottom-up and top-down attention for image captioning and visual question answering. InProceedings of the IEEE conference on computer vision and pattern recognition. 6077–6086
2018
-
[2]
Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation. InProceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), Steven Bethard, Marine Carpuat, Marianna Apidi- anaki, Saif M. Mohammad, Daniel Cer, and ...
-
[3]
Jiahe Chen, Jiaying He, Qian Shao, Qiyuan Chen, Jiahe Ying, Hongxia Xu, Jintai Chen, Jianwei Zheng, and Jian Wu. 2025. Mitigating Hallucination of Large Vision-Language Models via Dynamic Logits Calibration.arXiv preprint arXiv:2506.21509(2025)
arXiv 2025
-
[4]
Yixin Cheng, Hongcheng Guo, Yangming Li, and Leonid Sigal. 2025. Revealing Weaknesses in Text Watermarking Through Self-Information Rewrite Attacks. InForty-second International Conference on Machine Learning
2025
-
[5]
Miranda Christ, Sam Gunn, and Or Zamir. 2024. Undetectable watermarks for language models. InThe Thirty Seventh Annual Conference on Learning Theory. PMLR, 1125–1139
2024
-
[6]
Sumanth Dathathri, Abigail See, Sumedh Ghaisas, Po-Sen Huang, Rob McAdam, Johannes Welbl, Vandana Bachani, Alex Kaskasoli, Robert Stanforth, Tatiana Matejovicova, et al. 2024. Scalable watermarking for identifying large language model outputs.Nature634, 8035 (2024), 818–823
2024
-
[8]
Zhiwei He, Binglin Zhou, Hongkun Hao, Aiwei Liu, Xing Wang, Zhaopeng Tu, Zhuosheng Zhang, and Rui Wang. 2024. Can Watermarks Survive Translation? On the Cross-lingual Consistency of Text Watermark for Large Language Models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 4115–4129
2024
-
[9]
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The Curi- ous Case of Neural Text Degeneration. InInternational Conference on Learning Representations
2020
Show all 37 references
-
[10]
Nikola Jovanović, Robin Staab, and Martin Vechev. 2024. Watermark stealing in large language models. InProceedings of the 41st International Conference on Machine Learning. 22570–22593
2024
-
[11]
John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. 2023. A watermark for large language models. InInternational Conference on Machine Learning. PMLR, 17061–17084
2023
-
[12]
Gregory Kang Ruey Lau, Xinyuan Niu, Hieu Dao, Jiangwei Chen, Chuan-Sheng Foo, and Bryan Kian Hsiang Low. 2024. Waterfall: Scalable Framework for Robust Text Watermarking and Provenance for LLMs. InProceedings of the 2024 Con- ference on Empirical Methods in Natural Language Pr...
2024 doi
-
[13]
Taehyun Lee, Seokhee Hong, Jaewoo Ahn, Ilgee Hong, Hwaran Lee, Sangdoo Yun, Jamin Shin, and Gunhee Kim. 2024. Who wrote this code? watermarking for code generation. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers...
2024
-
[14]
Jiaming Li, Jiacheng Zhang, Zequn Jie, Lin Ma, and Guanbin Li. 2025. Mitigating hallucination for large vision language model by inter-modality correlation calibration decoding.arXiv preprint arXiv:2501.01926(2025)
2025 arXiv
-
[15]
Wenyan Li, Raphael Tang, Chengzu Li, Caiqi Zhang, Ivan Vulić, and Anders Søgaard. 2025. Lost in Embeddings: Information Loss in Vision–Language Models. InFindings of the Association for Computational Linguistics: EMNLP 2025, Christos Christodoulopoulos, Tanmoy Chakraborty, Car...
2025
-
[16]
Yue Li, Xin Yi, Dongsheng Shi, Yongyi Cui, Gerard de Melo, and Linlin Wang
-
[17]
Yue Li, Xin Yi, Dongsheng Shi, Gerard De Melo, Xiaoling Wang, and Linlin Wang. 2025. Hierarchical Safety Realignment: Lightweight Restoration of Safety in Pruned Large Vision-Language Models. InFindings of the Association for Computational Linguistics: ACL 2025, Wanxiang Che, ...
2025 doi
-
[18]
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. InEuropean conference on computer vision. Springer, 740–755
2014
-
[19]
Aiwei Liu, Leyi Pan, Yijian Lu, Jingjing Li, Xuming Hu, Xi Zhang, Lijie Wen, Irwin King, Hui Xiong, and Philip Yu. 2024. A survey of text watermarking in the era of large language models.Comput. Surveys57, 2 (2024), 1–36
2024
-
[20]
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024. LLaVA-NeXT: Improved reasoning, OCR, and world knowledge. https://llava-vl.github.io/blog/2024-01-30-llava-next/
2024
-
[21]
Shuliang Liu, Zheng Qi, Jesse Jiaxi Xu, Yibo Yan, Junyan Zhang, He Geng, Aiwei Liu, Peijie Jiang, Jia Liu, Yik-Cheung Tam, and Xuming Hu. 2025. VLA-Mark: A cross modal watermark for large vision-language alignment models. InProceed- ings of the 2025 Conference on Empirical Met...
2025 doi
-
[22]
Leyi Pan, Aiwei Liu, Zhiwei He, Zitian Gao, Xuandong Zhao, Yijian Lu, Binglin Zhou, Shuliang Liu, Xuming Hu, Lijie Wen, et al . 2024. MarkLLM: An Open- Source Toolkit for LLM Watermarking. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing...
2024
-
[23]
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a Method for Automatic Evaluation of Machine Translation. InProceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Pierre Isabelle, Eugene Charniak, and Dekang Lin (Eds.)...
2002 doi
-
[24]
Claude E Shannon. 1948. A mathematical theory of communication.The Bell system technical journal27, 3 (1948), 379–423
1948
-
[25]
Qwen Team. 2025. Qwen3 Technical Report. arXiv:2505.09388 [cs.CL] https: //arxiv.org/abs/2505.09388
2025 arXiv
-
[26]
Junyang Wang, Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Jiaqi Wang, Haiyang Xu, Ming Yan, Ji Zhang, et al. 2023. Amber: An llm-free multi-dimensional benchmark for mllms hallucination evaluation.arXiv preprint arXiv:2311.07397(2023)
2023 arXiv
-
[27]
Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. 2025. Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency.arXiv preprint arXiv:2508.18265(2025)
2025 arXiv
-
[28]
Yidan Wang, Yubing Ren, Yanan Cao, and Binxing Fang. 2025. From Trade-off to Synergy: A Versatile Symbiotic Watermarking Framework for Large Language Models. InProceedings of the 63rd Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), Wa...
2025 doi
-
[29]
Zongqi Wang, Tianle Gu, Baoyuan Wu, and Yujiu Yang. 2025. MorphMark: Flexible Adaptive Watermarking for Large Language Models. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Wanxiang Che, Joyce Nabende, Ekater...
2025 doi
-
[30]
Qilong Wu and Varun Chandrasekaran. 2024. Bypassing LLM Watermarks with Color-Aware Substitutions. InProceedings of the 62nd Annual Meeting of the Yue Li, Xin Yi, Dongsheng Shi, Yongyi Cui, Gerard de Melo, and Linlin Wang Association for Computational Linguistics (Volume 1: Lo...
2024
-
[31]
Mang Ye, Xuankun Rong, Wenke Huang, Bo Du, Nenghai Yu, and Dacheng Tao
-
[32]
Xin Yi, Yue Li, Dongsheng Shi, Linlin Wang, Xiaoling Wang, and Liang He. 2026. Latent-space adversarial training with post-aware calibration for defending large language models against jailbreak attacks.Expert Systems with Applications296 (2026), 129101. doi:10.1016/j.eswa.2025.129101
2026
-
[33]
A survey of safety on large vision-language models: Attacks, defenses and evaluations.arXiv preprint arXiv:2502.14881(2025)
2025 arXiv
-
[34]
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi
-
[35]
Xin Yi, Yue Li, Shunfan Zheng, Linlin Wang, Xiaoling Wang, and Liang He. 2025. Unified attacks to large language model watermarks: Spoofing and scrubbing in unauthorized knowledge distillation.Knowledge-Based Systems329 (2025), 114295. doi:10.1016/j.knosys.2025.114295
2025
-
[38]
Qi Zheng, Shuliang Liu, Yu Huang, Sihang Jia, Jungang Li, Lyuhao Chen, Junhao Chen, Hanqian Li, Aiwei Liu, Yibo Yan, et al. 2026. A Visual Semantic Adaptive Watermark grounded by Prefix-Tuning for Large Vision-Language Model.arXiv preprint arXiv:2601.07291(2026). A Experimenta...
2026
-
[2020]
InInternational Con- ference on Learning Representations
BERTScore: Evaluating Text Generation with BERT. InInternational Con- ference on Learning Representations
-
[2025]
From Injection to Defense: Constructing Edit-Based Fingerprints for Large Language Models.arXiv preprint arXiv:2509.03122(2025)
2025 arXiv
Reviewed August 3, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.