{"id":"c5133a91-5178-40de-a0ad-6567222b7362","arxiv_id":"2508.21072","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A competition-winning pipeline removes 95.7% of StegaStamp and TreeRing watermarks on the NeurIPS 2024 benchmark by combining VAE fine-tuning, diffusion purification, and translation tricks.","lead":"This paper describes the winning attack from the NeurIPS 2024 challenge that removes invisible watermarks from AI-generated images while keeping them looking nearly unchanged. It combines fine-tuned autoencoders, diffusion models, and color restoration to defeat two common watermarking methods.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"VAE is trained to map m-watermarked images to (1-m)-watermarked images, not to clean images; 'near-perfect removal' is actually message-flip evasion, so the watermark is not destroyed.","rationale":"The reader's weakest assumption concerns transferability of the paired-data VAE fine-tuning from the public WAVES StegaStamp to the challenge's modified StegaStamp. That is a legitimate concern, and the leaderboard result gives only indirect evidence of transfer. However, the more load-bearing issue is upstream: even assuming perfect transfer, the method as specified does not remove the watermark signal. The training target in Equation (1) is itself a StegaStamp-watermarked image (with message 1-m), so a converged VAE outputs a watermarked image, not a clean one. The paper's claims of 'watermark removal' and 'stripping away the watermark' are therefore not supported by the stated mechanism. This is not an external-consensus disagreement; it is an internal inconsistency between the loss objective and the interpretation. The challenge's detection metric may be fooled by message flipping, and the first-place leaderboard result is credible evidence of that specific form of evasion. But the abstract's 'near-perfect watermark removal with negligible impact on residual image quality' should be reframed as 'successful evasion of the challenge's message-matching detector by inverse-message embedding', and the paper should report decoder outputs to demonstrate that no residual watermark remains under a broader presence-detection criterion. Given the strong competition result, I would not reject the paper, but the central claim as written needs qualification and additional evidence; hence CONDITIONAL.","tokens_in":18060,"tokens_out":9037,"duration_ms":100603,"concrete_test":"Run the public WAVES StegaStamp decoder on the beige-box VAE outputs (or rerun Algorithm 1 on the public model with known messages m). For each output, compute the Hamming distance / bit error rate to m, to 1-m, and to random 100-bit messages. If outputs decode to 1-m with near-zero BER while BER to m is near 0.5, the attack is flipping messages, not removing watermarks. Additionally, compute the output's pixel distance to the clean image x versus to the inverse-watermarked image StegaStamp(x, 1-m); if it is closer to the latter, the VAE is not artifact suppression.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"In Section 3.1.1, Equation (1) trains the VAE to reconstruct x_i = StegaStamp(x, 1-m) from input x_w = StegaStamp(x, m). If the loss goes to zero, the output is exactly a StegaStamp-watermarked image carrying the inverse message. The paper's language ('effectively strip away the watermark', 'suppress watermark artifacts') is therefore internally inconsistent with its own objective: the VAE learns a message-flip transformation, not a denoising or artifact-removal map. This explains why perceptual quality is high: the output remains on the StegaStamp image manifold rather than being projected to a clean-image manifold. The leaderboard detection score measures whether the decoded message is far from the originally embedded message m; flipping to 1-m achieves that, but the output would still be flagged by any detector that checks for the presence of a valid StegaStamp message rather than checking the specific original message. No experiment in the paper decodes the attacked images to verify whether the residual StegaStamp signal has actually been removed. The abstract's 'near-perfect watermark removal' is thus an overstatement of what the method demonstrates: it is a message-evasion attack on the challenge's detection definition, not demonstrated watermark destruction.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents the authors' first-place solution to the NeurIPS 2024 Erasing the Invisible challenge. For the beige-box track, the method fine-tunes an SDXL VAE on pairs of images watermarked with complementary StegaStamp messages, then applies test-time optimization and CIELAB color/contrast transfer, and removes TreeRing watermarks with a 7-pixel spatial translation. For the black-box track, the 300 test images are manually clustered into four artifact classes, and cluster-specific diffusion-based purification with ChatGPT captions and tuned noise strengths is applied. The reported official leaderboard detection scores are 0.037 (beige-box) and 0.043 (black-box), and the paper claims near-perfect watermark removal (95.7%).","tokens_in":18418,"tokens_out":8794,"duration_ms":92017,"significance":"The official first-place result is externally measured on the competition leaderboard, giving the central empirical claim credibility. The paper also contains useful ablations, a clear description of the attack components, and an honest account of unsuccessful approaches. However, the paper's own text and equations reveal two load-bearing problems. First, the VAE in Eq. (1) is trained to map a StegaStamp image with message m to the same image with message 1-m, so it is a message-flip evasion attack, not demonstrated watermark destruction; no experiment checks whether the output still contains a valid StegaStamp message. Second, the method was selected and tuned using repeated leaderboard queries on the same 300 test images, and the manual clustering is test-specific, so the reported 95.7% is a test-set-tuned score without holdout validation. These issues limit the paper's scientific generality and need to be addressed before the claimed 'watermark removal' is accepted.","major_comments":[{"comment":"The training target in Eq. (1) is x_i = StegaStamp(x, 1-m), not a clean image. Thus the VAE learns a message-complementing map, not a watermark-removal map. The abstract's 'near-perfect watermark removal' and the text's 'effectively strip away the watermark' are inconsistent with this objective. The leaderboard detection score only checks distance from the original message m; it does not test whether the output still contains a valid StegaStamp message. No experiment decodes the attacked images. Please reframe the contribution as a message-evasion attack or add a decoding experiment showing that no valid message remains.","section":"3.1.1, Eq. (1), Algorithm 1"},{"comment":"The paper states that leaderboard queries were allowed up to five times per day, and Figure 9 says the diffusion model choice was made 'from leaderboard submissions'; Section 3.3 says hyperparameters (especially s) were tuned to balance removal and fidelity. The 300 test images were manually clustered and the cluster assignments are listed by index in Table 5, so the method is not an algorithm that can be applied to new images without manual inspection. No holdout validation, cross-validation, or error bars are reported. The reported 95.7% is therefore a test-set-tuned score, not evidence of generalizable watermark removal. Please add a validation split or explicitly label the result as a leaderboard score.","section":"3.2/3.3, Algorithm 3, Table 5, Figure 9"},{"comment":"Test-time optimization minimizes MSE, LPIPS, and SSIM against the original watermarked image x_w. This should encourage the output to retain the watermark, and Table 3 indeed shows the detection score worsens from 0.023 to 0.033 when this step is added. The text's claim that 'this step effectively removes the watermark signal' is not supported. Please revise the description or provide evidence that the step removes rather than re-introduces the watermark.","section":"3.1.1, Eq. (2), Algorithm 2, Table 3"},{"comment":"The beige-box attack is built on pairs generated with the public WAVES StegaStamp model, while the challenge uses a modified StegaStamp whose parameters are unknown. The paper does not directly verify that the public model's artifacts transfer to the challenge variant. Since the beige-box adversary knows the algorithm but not its hyperparameters, this assumption is load-bearing for the VAE attack and for the two black-box clusters that reuse the VAE pipeline. Please provide a transfer analysis, e.g., decoding rates on challenge images with the public model before and after the attack.","section":"3.1.1 'Paired Dataset Generation'"}],"minor_comments":[{"comment":"The cluster numbering in Algorithm 3 is inconsistent with Section 3.3 and Figure 5: Algorithm 3 maps Cluster 1 to boundary artifacts and Cluster 2 to circular Fourier patterns, whereas Section 3.3 lists Cluster 1 as no artifacts, Cluster 2 as boundary, Cluster 3 as circular, and Cluster 4 as square. This makes the pipeline ambiguous.","section":"Algorithm 3"},{"comment":"The Figure 2 caption says the VAE minimizes MSE loss 'effectively suppressing watermark artifacts', but Eq. (1) minimizes reconstruction to the inverse-message image. Rephrase to avoid implying clean-image reconstruction.","section":"Figure 2 and Section 3.1.1"},{"comment":"References [28] and [29] are the same paper; merge them.","section":"References"},{"comment":"No code or trained models are linked. Since the contribution is an attack pipeline with many engineering components, releasing code/models would materially improve reproducibility.","section":"Reproducibility"},{"comment":"Ours has lower PSNR than 'w/o Pixel Restoration' (14.353 vs. 14.878), while the text says restoration improves fidelity. Clarify that PSNR is lower because of the global shift and that the improvement claim refers to perceptual and distribution metrics.","section":"Table 6"},{"comment":"Typos: 'Invisible W atermark' in the title, 'invese' in Figure 2, and 'oundary' in the Figure 4 caption.","section":"Various"}],"recommendation":"major_revision","confidential_remarks":"The leaderboard result is credible, and the paper is a useful competition report. The main risks are the overclaiming of watermark removal and the absence of validation beyond the test-set-tuned leaderboard. These are fixable with reframing and additional experiments, so major revision rather than rejection is appropriate. If the journal does not typically publish competition solution reports, the fit should also be considered by the editors."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid competition write-up whose main result — first place on both NeurIPS 2024 tracks — is externally validated on a public leaderboard. That is real evidence. The ablation tables are clean, the failed-attempts section is honest, and the cluster-specific design is a sensible way to handle heterogeneous black-box watermarks. For anyone working on watermark robustness, this is the reference attack configuration from the challenge.\n\nNow the soft spots, in proportion. The biggest one is the stress-test concern about the VAE objective. Equation (1) trains the VAE to map StegaStamp(x, m) to StegaStamp(x, 1-m), not to a clean image. So the output is a watermarked image carrying the inverse message. That is message evasion, not watermark destruction. The paper says it 'effectively strip[s] away the watermark' and claims 'near-perfect watermark removal,' but those statements are internally inconsistent with the training objective. The leaderboard detection metric only measures distance from the original message, so flipping works under that metric. But the paper never decodes the attacked images to show the residual StegaStamp signal is actually gone. If the threat model is 'this image should not carry a detectable watermark at all,' the beige-box attack likely fails. The abstract's claim is too strong as written.\n\nSecond, the hyperparameters were tuned through up to five leaderboard queries per day on the test set. The manual clustering is also explicitly test-specific — they inspected the 300 images and grouped them by artifacts. So the headline numbers are real for this benchmark, but they almost certainly overstate what a general-purpose black-box attacker would achieve on unseen images, since the clustering and per-cluster settings are fitted to these 300 images. There are no error bars and the method is only applied once. That said, this is a competition paper; some test-set tuning is the norm. The authors are transparent about the queries, which is to their credit.\n\nThird, the diffusions and VAE components are mostly borrowed from cited work. The novelty is the combination and the cluster-specific tuning, not any new principle. That is fine for a solution paper; nobody should expect a new theory in a competition write-up.\n\nWho is this for? Anyone building watermark-removal attacks or defenses will want to know how the winning entry worked, especially the TreeRing translation trick and the cluster-wise diffusion strengths. It deserves a serious referee because the empirical result is benchmark-defining, but the referee should push hard on the message-flip versus removal distinction and ask for a post-hoc decode experiment on the attacked images.\n\nRecommendation: send it to peer review, conditional on clearly recharacterizing the VAE attack as message substitution and softening 'removal' claims accordingly.","headline":"First place on both tracks is real and externally measured, but the headline claim of 'watermark removal' is overstated because the core VAE attack flips the message rather than destroying it.","tokens_in":18842,"tokens_out":1381,"would_cite":true,"duration_ms":17154,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper shows that adaptive attacks can strip invisible watermarks from images while keeping them visually near-identical, defeating both a known-algorithm and an unknown-algorithm detector at the 0.1% false-positive threshold.","keywords":["image watermark removal","adversarial attack","StegaStamp","TreeRing watermark","variational autoencoder","diffusion purification","NeurIPS 2024 challenge","watermark robustness"],"falsifier":"Generate a new test set with a StegaStamp variant whose encoder uses a message-dependent key, so that the same image watermarked with m and with 1−m differ not only in the additive watermark component but also in the transformation itself; run the paired-VAE attack and measure the detection score at the 0.1% false-positive threshold. If the detection score stays near 1.0 while image-quality metrics remain unchanged, the central separability assumption is false.","tokens_in":17987,"feed_emoji":"🖼️","tokens_out":8592,"duration_ms":79982,"temperature":0.7,"pith_summary":"The paper demonstrates that invisible image watermarks can be removed with near-perfect success while preserving visual quality, under two realistic threat models: a beige-box setting where the watermarking algorithm is known but its parameters are not, and a black-box setting where the algorithm is unknown. For known StegaStamp-style watermarks, the key idea is to fine-tune a variational autoencoder on pairs of the same image watermarked with a message and its inverse, training the VAE to cancel the message-dependent signal. For unknown watermarks, the paper clusters images by spatial or spectral artifact signatures and applies a tailored combination of diffusion-based purification, VAE cancellation, and a small spatial translation. On the NeurIPS 2024 challenge, the attack lowered the detection scores to 0.037 and 0.043 at a 0.1% false-positive rate, equivalent to roughly 95.7% watermark removal, while keeping quality metrics competitive. The paper argues these results show current watermarking schemes are not robust against adaptive, computationally feasible adversaries.","feed_headline":"Invisible watermarks erased in 95.7% of test images","feed_subtitle":"A NeurIPS-winning attack pairs message inversion with diffusion to strip StegaStamp and TreeRing marks while keeping images intact.","key_machinery":"The paired-message VAE fine-tuning is the central object: a variational autoencoder (an encoder/decoder pair) trained with an MSE loss to map a StegaStamp-watermarked image carrying message m to the same image watermarked with the inverted message 1−m. The inversion forces the latent bottleneck to drop the message-dependent signal, so on unseen images the fine-tuned VAE outputs a watermark-free reconstruction. Around this core, the paper adds test-time optimization (refining the VAE on the specific input using MSE + LPIPS + SSIM losses) and a CIELAB color/contrast transfer to restore fidelity, plus a spatial translation for Fourier-phase watermarks and a cluster-conditioned diffusion purific","core_discovery":"On the NeurIPS 2024 challenge's two tracks, the paper's attack lowers detection scores to 0.037 (beige-box) and 0.043 (black-box) at a 0.1% false-positive rate, meaning more than 95% of watermarked images pass as clean. The key technical move is a paired-message VAE: the adversary fine-tunes a pretrained variational autoencoder on pairs of the same image watermarked with a random 100-bit message and its complement, training it to reconstruct the complement image from the original. Because the only difference between the pair is the message, the VAE learns to discard the message-dependent watermark component and re-synthesize the content. After a test-time optimization step that aligns the re","pith_inferences":["The inverse-message cancellation trick should generalize to any learned watermark whose encoder is roughly symmetric under logical message inversion; testing it against other message-embedding schemes would be a direct extension.","Attack traces such as the 7-pixel translation or diffusion-induced smoothing could themselves be fingerprinted; a defender might build a detector that recognizes these removal artifacts and flags images as 'cleaned'.","The CIELAB color/contrast restoration is a model-agnostic quality booster that could be bolted onto other watermark-removal pipelines, potentially raising their quality scores without sacrificing removal rate.","Because the clustering step was based on 300 images and manual inspection, a larger challenge with more diverse watermarks could shift the artifact types; an automated artifact classifier would be needed to scale this approach."],"forward_implications":["At the challenge's 0.1% false-positive threshold, both detectors flag fewer than 5% of attacked images, so an attacker can strip the watermark from roughly 19 of every 20 images.","The StegaStamp VAE attack transfers from a paired training set built with a public StegaStamp encoder to the challenge's modified StegaStamp variant without retraining on challenge images.","TreeRing-style Fourier-phase watermarks are broken by a trivial 7-pixel spatial translation, with boundary columns restored to hide the shift; no model or training is needed.","The black-box attack is effective only because images are clustered by artifact signatures; a single global diffusion setting would not erase all four watermark families.","The success of these attacks implies watermark robustness claims should be tested under adaptive, cluster-aware adversaries rather than fixed corruption sets."],"supporting_citations":[{"why":"Defines the benchmark, the beige-box and black-box threat models, the 300-image test set, and the detection/quality metrics the attack optimizes against.","marker":"[10]"},{"why":"Describes StegaStamp, the learned watermarking method that the beige-box attack targets and uses to generate paired training images.","marker":"[53]"},{"why":"Describes TreeRing, the Fourier-phase watermark whose vulnerability to spatial translation the paper exploits.","marker":"[58]"},{"why":"Supplies the pretrained StegaStamp model and robustness-evaluation context used to build the paired dataset for VAE fine-tuning.","marker":"[4]"},{"why":"Provides the image-to-image diffusion process (forward noise injection and reverse denoising) used as the black-box purification engine.","marker":"[35]"},{"why":"Supplies the language-model-generated captions that condition the diffusion purification, preserving semantic content during regeneration.","marker":"[2]"}],"fun_headline_variants":["Win NeurIPS watermark removal: 95.7% erased","VAE attack strips invisible watermarks 95.7%","Adaptive VAE beats watermarking at NeurIPS 2024","95.7% watermark removal via paired VAE attack"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The beige-box attack relies on the assumption that the challenge's StegaStamp variant is close enough to the public StegaStamp model used to generate paired training data, and that fine-tuning on (message, inverse-message) pairs cancels the watermark on unseen images; if the actual encoder is substantially different or the message signal is not separable this way, the VAE cancellation would not transfer.","fun_headline_variants_meta":{"raw":{"variants":["Win NeurIPS watermark removal: 95.7% erased","VAE attack strips invisible watermarks 95.7%","Adaptive VAE beats watermarking at NeurIPS 2024","95.7% watermark removal via paired VAE attack"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000187,"raw_usage":{"total_tokens":1170,"prompt_tokens":752,"completion_tokens":418,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":496,"completion_tokens_details":{"reasoning_tokens":356}},"tokens_in":496,"tokens_out":418,"duration_ms":4325,"temperature":1.0,"reasoning_tokens":356,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T14:34:42.616858+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Generate a new test set with a StegaStamp variant whose encoder uses a message-dependent key, so that the same image watermarked with m and with 1−m differ not only in the additive watermark component but also in the transformation itself; run the paired-VAE attack and measure the detection score at the 0.1% false-positive threshold. If the detection score stays near 1.0 while image-quality metrics remain unchanged, the central separability assumption is false.","supporting_citations":[{"cited_title":"Stegastamp: Invisible hyperlinks in physical photographs","cited_arxiv_id":null,"evidence_quote":"Describes StegaStamp, the learned watermarking method that the beige-box attack targets and uses to generate paired training images."},{"cited_title":"Tree-rings wa- termarks: Invisible fingerprints for diffusion images.Advances in Neural Information Processing Systems, 36, 2024","cited_arxiv_id":null,"evidence_quote":"Describes TreeRing, the Fourier-phase watermark whose vulnerability to spatial translation the paper exploits."},{"cited_title":"Waves: Benchmarking the robustness of image watermarks","cited_arxiv_id":null,"evidence_quote":"Supplies the pretrained StegaStamp model and robustness-evaluation context used to build the paired dataset for VAE fine-tuning."}],"review_version":1}