REVIEW 4 major objections 5 minor 1 cited by
VideoMarkBench: Benchmarking Robustness of Video Watermarking
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read VideoMarkBench claims that existing video watermarking methods are not robust: white-box attackers can make false-negative and false-positive rates reach 1 with small bounded perturbations, and black-box attackers can remove watermarks…
desk verdict First systematic video watermarking robustness benchmark; the white-box breakage is real, but the unverified bit-independence threshold model and narrow black-box subset need fixing before the headline numbers are quoted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the bitwise-accuracy (BA) decision rule: a decoder outputs a bitstring per frame or per 8-frame group, and a video is declared watermarked when the fraction of bits matching the ground-truth watermark exceeds a threshold $\tau$; for 96-bit watermarks $\tau=67/96$ and for 32-bit $\tau=27/32$, chosen so the theoretical false-positive rate stays below $10^{-4}$ under a binomial model. White-box PGD attacks minimize the distance between decoded logits and the target watermark, score-based Square Attack and label-based Triangle Attack iteratively query the detector, and seven aggregation strategies determine how per-frame evidence is combined into a video-level verdict. The comparison of those aggregation strategies is what converts per-frame robustness results into the FNR and FPR numbers that support the paper's conclusion.
What would settle it
Run each decoder on a large set of never-watermarked videos and compare decoded bitstrings against random ground-truth watermarks; if the empirical match rate deviates from 0.5 or bits are correlated, the binomial threshold model in Appendix A.4 does not justify the claimed FPR below 0.01% and the attack-success numbers would have to be re-derived.
Extended reading notes
Core claim
On its own terms, the paper establishes that four current video watermarking methods—REVMark, StegaStamp, VideoSeal, and VideoShield—detect accurately when no perturbation is applied, with FNR and FPR near zero. Under white-box attacks, projected gradient descent with an $\ell_\infty$ bound as small as 0.001 drives both FNR and FPR to 1 for the three post-generation methods; forgery requires even smaller perturbations than removal, since the decoders were adversarially trained against removal but not forgery. Under black-box attacks, score-based Square Attack and label-based Triangle Attack remove watermarks within a few hundred queries, while forgery largely fails. Under common perturbations, JPEG and MPEG-4 compression and cropping remove watermarks once visual quality degrades, whereas blurring and frame-level manipulations leave them intact. The paper concludes that current video watermarks are not a reliable defense against a knowledgeable attacker.
Load-bearing premise
The load-bearing premise is that an unwatermarked video's decoded bits match a random ground-truth watermark with probability 0.5 independently, which is what certifies that the detection thresholds deliver a false-positive rate below 0.01%; if real decoders are biased or correlated on unwatermarked content, the reported FNR, FPR, and the broken conclusion shift.
Editorial extensions
If this is right
- White-box attackers can drive FNR and FPR to 1 with $\ell_\infty$ perturbations as small as 0.001, so a leaked or publicly available detector makes both removal and forgery trivial.
- Black-box removal succeeds within a few hundred queries, meaning API access alone is enough to strip watermarks from targeted videos.
- Aggregation strategy choices are not neutral: logit-level and median-based aggregation outperform BA-level and mean-based strategies, and detection-threshold trades removal robustness for forgery vulnerability.
- Common manipulations such as MPEG-4 compression and cropping can remove watermarks once they degrade visual quality, so ordinary video processing is a realistic removal risk.
- Current watermarking methods should incorporate temporal information across frames rather than relying on frame-level decisions if they are to resist these attacks.
Reading between the lines
- Editorial inference: a defender could combine median-based aggregation for forgery resistance with a detection-threshold rule for removal resistance, since the two failure modes favor opposite strategies.
- Editorial inference: the binomial-threshold assumption is empirically testable; if decoders turn out to be biased on unwatermarked content, thresholds would need per-method recalibration and the reported FPRs would change.
- Editorial inference: the white-box 'all methods broken' claim currently excludes VideoShield, which was omitted because of compute cost, so it rests on the three post-generation methods.
- Editorial inference: because black-box forgery largely failed while removal succeeded, the asymmetry suggests detection APIs leak enough signal for removal but not enough to synthesize a valid watermark.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces VideoMarkBench, a benchmark for evaluating the robustness of video watermarking methods under watermark removal and watermark forgery attacks. It constructs a new AI-generated video dataset (VideoMarkData) from Stable Video Diffusion, Sora, and Hunyuan Video across three visual styles, and evaluates four watermarking methods (REVMark, StegaStamp, VideoSeal, VideoShield) under white-box, black-box, and no-box threat models. It additionally proposes seven frame-level aggregation strategies. The main findings are that existing methods are accurate without perturbations, but white-box attacks drive both FNR and FPR to 1 with small ℓ∞-norm perturbations; black-box removal attacks succeed within a few hundred queries; and no-box common perturbations such as JPEG/MPEG-4 compression and cropping can remove watermarks. Watermark forgery is reported to be difficult in black-box and no-box settings. The paper releases code and data.
Significance. If the findings hold, this is a useful benchmark for a practically important problem. The paper covers a broad design space: three generative models, multiple watermarking methods, seven aggregation strategies, and three threat models, with both removal and forgery goals. Releasing code and data is a concrete strength that supports reproducibility. The theoretical FPR derivation in Appendix A.4 is standard and clearly presented, and the evaluation is genuinely empirical rather than circular. The main value is in the systematic comparison of aggregation strategies and the unified dataset, which the community can reuse. However, as detailed in the major comments, the quantitative conclusions currently rest on an unvalidated statistical model for decoder behavior and on attack descriptions whose stated objectives are inverted; these issues need to be resolved before the benchmark's headline claims can be fully trusted.
major comments (4)
- [Section 3.1, Eq. (1) and the subset-of-frames objective] The stated white-box attack objectives appear to be inverted relative to the attack goals. In Eq. (1), the removal attack is written as minimizing a loss l(Dec(I+δ), wg) where l 'measures the distance between two vectors'; minimizing a distance toward wg would push the decoded watermark toward the ground truth, which is the opposite of removal. The text then says forgery 'reverses' the objective to maximize the loss, which would push the output away from wg, again opposite. Similarly, the second-scenario removal objective, minδ −Σ_i sign(wg_i − 0.5)·Dec(I+δ)_i, as written drives logits toward the ground-truth bits (high for wg=1, low for wg=0), i.e., it strengthens the watermark rather than removing it. The reported results (FNR=1 in Figures 2 and 4) show that the actual experiments must have used the opposite sign convention, so the written definitions do not match the implementation and the attacks are not reproducible from the paper. Please correct the equations, define the loss function precisely (e.g., cross-entropy or negative bitwise accuracy), and confirm that the released code uses the same objective as the corrected text.
- [Appendix A.4 and Table 6] The statistical model that sets τ=67/96, τ=27/32, and the per-frame false-positive probability P used to choose k for detection-threshold aggregation is load-bearing but unverified. The derivation assumes that on unwatermarked content the decoded watermark bits are independent of the ground-truth watermark and each bit matches with probability 0.5. If real decoders produce biased or correlated bits on unwatermarked videos, the actual FPR at these thresholds is not the claimed <0.01%, and the white-box forgery results (FPR=1) and the aggregation-strategy rankings could partly be artifacts of miscalibrated thresholds. The fact that Table 6 reports FPR=0.000 on 1,000 Kinetics videos does not validate the tail: under the model the expected number of false positives is below 0.1, so observing zero is consistent with a true FPR several times larger. Please report the empirical distribution of bitwise accuracy on a large set of unwatermarked videos (e.g., all frames of 10,000+ videos), test the independence and unbiasedness assumptions, and give confidence intervals for the empirical FPR at the chosen thresholds. If the calibrated thresholds differ, re-run the white-box forgery and detection-threshold aggregation experiments with the corrected thresholds.
- [Section 5.3, Figures 5 and 6] The scope of the black-box removal evaluation is stated inconsistently. The text says 'by default, we only evaluate on videos generated by SVD and realistic style, using BA-mean aggregation,' yet Figure 5c–d and the accompanying bullets report comparisons across generative models (SVD, Sora, Hunyuan) and styles (realistic, cartoon, sci-fi), and Figure 6 reports Triangle Attack results across the same dimensions. If the model/style comparisons were run on the full 40-videos-per-model-and-style subset, that contradicts the 'only SVD realistic' default; if they were not, the general black-box conclusions are extrapolated from a narrow subset. Please clarify exactly which videos, how many per condition, and which aggregation strategies were used for each panel in Figures 5 and 6, and report standard errors or confidence bands. Also clarify how many queries were allowed and whether the same video subset was used for Square and Triangle attacks.
- [Appendix A.4, FPR formula] There is an off-by-one inconsistency in the theoretical FPR formula. The text defines FPR(τ)=Pr(BA(w,wg)>τ) but then sums from k=⌈nτ⌉, which corresponds to BA≥τ (when nτ is an integer, e.g., 96·67/96=67). Since the detection rule in Equation (4) uses ≥τ, the FPR expression should be Pr(BA≥τ) or, if strict inequality is intended, the sum should start at ⌈nτ⌉+1. This changes the numerical FPR guarantee slightly and should be corrected for consistency.
minor comments (5)
- [Abstract and Section 5.2.1] The abstract and introduction claim the benchmark covers 'four watermarking methods' under all settings, but VideoShield is excluded from white-box and black-box evaluations (Sections 5.2 and 5.3), and in the no-box setting VideoShield is evaluated only on SVD videos. Please make these exclusions explicit in the contributions and in the relevant figure captions.
- [Figure 13] The caption of Figure 13 says the figure shows results for 'different video watermarking methods,' but subfigure (d) uses the legend 'Realistic, Cartoon, Sci-fi' and reports style-level results rather than method-level results. Please correct the caption or the subfigure so that the reported quantities match the legend.
- [Figure 1 and Section 2.1] There are small typos: 'BA lavel' should be 'BA level' in Figure 1, and 'Standard Derivation' in several figure captions should be 'Standard Deviation.'
- [Table 3] The time costs in Table 3 would be easier to read if the units and the exact configuration (GPU, batch size) were specified; the current formatting with scientific notation is also cluttered.
- [Section 5.2.1 and Appendix A.5] The t-tests for differences across models and styles are reported with p-values of approximately 0.038 and 0.029, but it is not stated how many samples were used, whether the tests are paired, or whether multiple comparisons were considered. Please add these details.
Circularity Check
No significant circularity: the robustness conclusions are empirical measurements against actual detectors, and the self-cited threshold guarantee is re-derived in the appendix.
full rationale
The paper's central claims are empirical measurements. White-box removal and forgery results in Section 5.2.1 are obtained by running PGD directly against the actual watermark decoders (Equation 1), and black-box results in Section 5.3 come from querying the detection API with Square Attack and Triangle Attack. These conclusions do not reduce to a fitted parameter or to the authors' prior work. The detection thresholds tau=67/96 and 27/32 are taken from the original watermarking methods and are re-derived in Appendix A.4 from a stated binomial model: 'for an unwatermarked image, the decoded watermark w is independent of wg, and each bit matches with probability 0.5.' This is a modeling assumption, not a fit to the benchmark data, and the paper does not present the clean FPR of 0.000 on 1,000 Kinetics videos as proof of the theoretical 0.01% bound. The self-citation of Jiang et al. 2023 [14] in the threshold-selection paragraph is not load-bearing because Appendix A.4 supplies the full derivation of FPR(tau). The other self-citations [11,12,13] are background attack references and are not used to justify the benchmark's findings. The Appendix A.4 bit-independence model is a potential calibration risk, but it would not force the central conclusions: for removal, a lower (more permissive) threshold makes removal harder, yet FNR still reaches 1; for forgery, the white-box attack drives bitwise accuracy to 1, far above any plausible threshold, so the FPR=1 result is not an artifact of a too-low threshold. The aggregation-strategy comparisons are also empirical and mechanistic, not derived from the paper's own fitted quantities. Overall, the derivation chain is self-contained against external detectors and standard attack algorithms.
Assumptions & free parameters
assumptions (4)
- domain assumption Decoded watermark bits for unwatermarked content are independent of the ground-truth watermark and each bit matches with probability 0.5.
- domain assumption The white-box attacker has full access to detector parameters and can backpropagate through the decoder.
- domain assumption The extension of image-level Square Attack and Triangle Attack to video by flattening frames is a faithful adaptation.
- domain assumption SVD, Sora, and Hunyuan Video outputs are representative of AI-generated videos.
Cite this review
Pith. "Pith review of VideoMarkBench: Benchmarking Robustness of Video Watermarking." pith.science (2026). https://pith.science/paper/LHTR6MU7
@misc{pith2026250521620,
author = {Pith},
title = {Pith review of: VideoMarkBench: Benchmarking Robustness of Video Watermarking},
year = {2026},
howpublished = {\url{https://pith.science/paper/LHTR6MU7}},
note = {Machine review of arXiv:2505.21620}
}
read the original abstract
The rapid development of video generative models has led to a surge in highly realistic synthetic videos, raising ethical concerns related to disinformation and copyright infringement. Recently, video watermarking has been proposed as a mitigation strategy by embedding invisible marks into AI-generated videos to enable subsequent detection. However, the robustness of existing video watermarking methods against both common and adversarial perturbations remains underexplored. In this work, we introduce VideoMarkBench, the first systematic benchmark designed to evaluate the robustness of video watermarks under watermark removal and watermark forgery attacks. Our study encompasses a unified dataset generated by three state-of-the-art video generative models, across three video styles, incorporating four watermarking methods and seven aggregation strategies used during detection. We comprehensively evaluate 12 types of perturbations under white-box, black-box, and no-box threat models. Our findings reveal significant vulnerabilities in current watermarking approaches and highlight the urgent need for more robust solutions. Our code is available at https://github.com/zhengyuan-jiang/VideoMarkBench.
Figures
Figures from the paper (20 more)
Forward citations
Cited by 1 Pith paper
-
Who Gets Flagged? The Pluralistic Evaluation Gap in AI Content Watermarking
Major watermarking benchmarks omit cross-lingual, cultural, and demographic reporting, creating a pluralistic evaluation gap that current governance mandates ignore.
Reference graph
Works this paper leans on
-
[1]
Deepfake video of zelenskyy could be ’tip of the iceberg’ in info war
Bobby Allyn. Deepfake video of zelenskyy could be ’tip of the iceberg’ in info war. https://www.npr.org/2022/03/16/1087062648/deepfake-video-zelenskyy-experts-war-manipulation- ukraine-russia. Online; accessed March 16, 2022
work page 2022
-
[2]
Waves: Benchmarking the robustness of image watermarks
Bang An, Mucong Ding, Tahseen Rabbani, Aakriti Agrawal, Yuancheng Xu, Chenghao Deng, Sicheng Zhu, Abdirisak Mohamed, Yuxin Wen, Tom Goldstein, et al. Waves: Benchmarking the robustness of image watermarks. In International Conference on Machine Learning, 2024
work page 2024
-
[3]
Square attack: a query-efficient black-box adversarial attack via random search
Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square attack: a query-efficient black-box adversarial attack via random search. In European Conference on Computer Vision, 2020
work page 2020
-
[4]
Securing the future of genai: Policy and technology
Mihai Christodorescu, Ryan Craven, Soheil Feizi, Neil Gong, Mia Hoffmann, Somesh Jha, Zhengyuan Jiang, Mehrdad Saberi Kamarposhti, John Mitchell, Jessica Newman, et al. Securing the future of genai: Policy and technology. arXiv, 2024
work page 2024
-
[5]
Learning temporal coherence via self-supervision for gan-based video generation
Mengyu Chu, You Xie, Jonas Mayer, Laura Leal-Taixé, and Nils Thuerey. Learning temporal coherence via self-supervision for gan-based video generation. ACM Transactions on Graphics (TOG), 2020. 11
work page 2020
-
[6]
Diffusion models beat gans on image synthesis
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Conference on Neural Information Processing Systems, 2021
work page 2021
-
[7]
Video seal: Open and efficient video watermarking
Pierre Fernandez, Hady Elsahar, I Zeki Yalniz, and Alexandre Mourachko. Video seal: Open and efficient video watermarking. arXiv, 2024
work page 2024
-
[8]
Explaining and harnessing adversarial examples
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations, 2015
work page 2015
Show all 32 references
-
[9]
Image quality metrics: Psnr vs
Alain Hore and Djemel Ziou. Image quality metrics: Psnr vs. ssim. In International Conference on Pattern Recognition, 2010
2010
-
[10]
Videoshield: Regulating diffusion-based video generation models via watermarking
Runyi Hu, Jie Zhang, Yiming Li, Jiwei Li, Qing Guo, Han Qiu, and Tianwei Zhang. Videoshield: Regulating diffusion-based video generation models via watermarking. In International Conference on Learning Representations, 2025
2025
-
[11]
Stable signature is unstable: Removing image watermark from diffusion models
Yuepeng Hu, Zhengyuan Jiang, Moyang Guo, and Neil Gong. Stable signature is unstable: Removing image watermark from diffusion models. arXiv, 2024
2024
-
[12]
A transfer attack to image watermarks
Yuepeng Hu, Zhengyuan Jiang, Moyang Guo, and Neil Gong. A transfer attack to image watermarks. In International Conference on Learning Representations, 2025
2025
-
[13]
Watermark-based detection and attribution of ai-generated content
Zhengyuan Jiang, Moyang Guo, Yuepeng Hu, and Neil Zhenqiang Gong. Watermark-based detection and attribution of ai-generated content. arXiv, 2024
2024
-
[14]
Evading watermark based detection of ai-generated content
Zhengyuan Jiang, Jinghuai Zhang, and Neil Zhenqiang Gong. Evading watermark based detection of ai-generated content. In ACM SIGSAC Conference on Computer and Communications Security, 2023
2023
-
[15]
The kinetics human action video dataset
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. The kinetics human action video dataset. arXiv, 2017
2017
-
[16]
Hunyuanvideo: A systematic framework for large video generative models
Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv, 2024
2024
-
[17]
Leveraging optimization for adaptive attacks on image watermarks
Nils Lukas, Abdulrahman Diaa, Lucas Fenaux, and Florian Kerschbaum. Leveraging optimization for adaptive attacks on image watermarks. In International Conference on Learning Representations, 2024
2024
-
[18]
To- wards deep learning models resistant to adversarial attacks
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. To- wards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018
2018
-
[19]
Diffusion models for adversarial purification
Weili Nie, Brandon Guo, Yujia Huang, Chaowei Xiao, Arash Vahdat, and Animashree Anandkumar. Diffusion models for adversarial purification. In International Conference on Machine Learning, 2022
2022
-
[20]
OpenAI. Gpt-4. https://chatgpt.com/. Online; accessed March 14, 2023
2023
-
[21]
OpenAI. Sora. https://sora.chatgpt.com/explore. Online; accessed November 21, 2023
2023
-
[22]
An efficient method for finding the minimum of a function of several variables without calculating derivatives
Michael JD Powell. An efficient method for finding the minimum of a function of several variables without calculating derivatives. The Computer Journal, 1964. 12
1964
-
[23]
Robustness of ai-image detectors: Fundamental limits and practical attacks
Mehrdad Saberi, Vinu Sankar Sadasivan, Keivan Rezaei, Aounon Kumar, Atoosa Chegini, Wenxiao Wang, and Soheil Feizi. Robustness of ai-image detectors: Fundamental limits and practical attacks. In International Conference on Learning Representations, 2024
2024
-
[24]
Stable video diffusion
Stability-AI. Stable video diffusion. https://github.com/Stability-AI/generative-models. GitHub; accessed November 21, 2023
2023
-
[25]
Intriguing properties of neural networks
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations, 2014
2014
-
[26]
Stegastamp: Invisible hyperlinks in physical photographs
Matthew Tancik, Ben Mildenhall, and Ren Ng. Stegastamp: Invisible hyperlinks in physical photographs. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020
2020
-
[27]
Triangle attack: A query-efficient decision-based adversarial attack
Xiaosen Wang, Zeliang Zhang, Kangheng Tong, Dihong Gong, Kun He, Zhifeng Li, and Wei Liu. Triangle attack: A query-efficient decision-based adversarial attack. In European Conference on Computer Vision, 2022
2022
-
[28]
Image quality assessment: from error visibility to structural similarity
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 2004
2004
-
[29]
Tree-ring watermarks: Fingerprints for diffusion images that are invisible and robust
Yuxin Wen, John Kirchenbauer, Jonas Geiping, and Tom Goldstein. Tree-ring watermarks: Fingerprints for diffusion images that are invisible and robust. In Conference on Neural Information Processing Systems, 2023
2023
-
[30]
A novel deep video watermarking framework with enhanced robustness to h
Yulin Zhang, Jiangqun Ni, Wenkang Su, and Xin Liao. A novel deep video watermarking framework with enhanced robustness to h. 264/avc compression. In ACM International Conference on Multimedia, 2023
2023
-
[31]
Invisible image watermarks are provably removable using generative ai
Xuandong Zhao, Kexun Zhang, Zihao Su, Saastha Vasan Vasan, Ilya Grishchenko, Christopher Kruegel, Giovanni Vigna, Yu-Xiang Wang, and Lei Li. Invisible image watermarks are provably removable using generative ai. In Conference on Neural Information Processing Systems, 2024
2024
-
[32]
watermarked
Jiren Zhu, Russell Kaplan, Justin Johnson, and Li Fei-Fei. Hidden: Hiding data with deep networks. In European Conference on Computer Vision, 2018. 13 A Appendix A.1 Experiments Compute Resources We conduct our experiments on 18 NVIDIA-RTX-6000 GPUs, each with 24 GB memory. Th...
2018
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.