{"id":"8f1bdc8e-1357-48b9-9422-b1117be7f533","arxiv_id":"2412.02576","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Transfer-based no-box watermark evasion largely fails without aligned surrogate models, and a simple one-surrogate perturbation (OFT) matches or exceeds the expensive optimization-based attack in 11 of 12 tested configurations.","lead":"This paper tests whether transfer-based no-box attacks on image watermarks work under realistic assumptions, and finds they mostly fail unless the attacker knows and mirrors the victim model setup. It also proposes a much cheaper attack, OFT, that uses one surrogate model and matches or beats the expensive optimization-based attack in most tested configurations.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper tests Hu et al.'s attack only with HiDDeN surrogates, omitting the multi-architecture surrogate pool Hu et al. used; the 'unaligned transfer fails' conclusion may not apply to the original attack.","rationale":"Both the abstract's 'failed attack, with an evasion rate at most 21.1%' and the conclusion that no-box transfer is challenging rest on §6.2's negative result. That result was obtained with a HiDDeN-only surrogate pool. Hu et al.'s published cross-method experiment, which the paper acknowledges, used a mixed HiDDeN+StegaStamp pool; its success on StegaStamp and Stable Signature may come from having a matching surrogate in the pool, not from transfer across architectures. The paper's setup removes that possibility by construction, so it cannot support a general statement about Hu et al.'s attack. This is an internal mismatch between the claimed scope and the experimental design, not a disagreement with existing consensus. The reader's weakest assumption identifies the same gap, so I agree. The correct disposition is unchanged from the reader's CONDITIONAL: the paper should be accepted only after the missing baseline is run. If the diverse-pool experiment shows high evasion on non-HiDDeN targets, the headline claim would need to be substantially weakened; if it confirms failure, the paper's conclusion would be supported.","tokens_in":18662,"tokens_out":6845,"duration_ms":65628,"concrete_test":"Run Hu et al.'s attack with the original multi-architecture surrogate pool (HiDDeN and StegaStamp, as in Hu et al. Fig. 6) against target StegaStamp, MBRS, RivaGAN, and HiDDeN, using the same k values (e.g., 10, 50), ℓ∞=0.25 budget, evaluation set, and detection threshold as in §6.2, and report evasion rates per target. If StegaStamp (or any non-HiDDeN target) is evaded at a rate far above 21.1%, the paper's conclusion that unaligned transfer fails does not hold for Hu et al.'s attack as designed. As a minimal variant, repeat with surrogate pools of 2–4 families (HiDDeN, StegaStamp, MBRS, RivaGAN) to test robustness to unknown target identity.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 6.2 evaluates Hu et al.'s transfer attack on target MBRS, RivaGAN, StegaStamp, and HiDDeN using only HiDDeN surrogate models. The paper itself notes (Sec. 6.2) that Hu et al.'s analogous cross-method experiment included both HiDDeN and StegaStamp in the surrogate pool, with targets including vanilla/smoothed HiDDeN, StegaStamp, and Stable Signature. By restricting surrogates to one family, the paper tests a strictly weaker version of the attack than the one proposed in Hu et al. The central negative claim—relaxing the alignment assumption gives at most 21.1% evasion—therefore depends on an assumption about the original attack that is not tested: that including a diverse surrogate pool (e.g., HiDDeN plus StegaStamp) would not restore transferability to StegaStamp or other targets. A no-box attacker who does not know the victim method can still include all public watermarking families in the surrogate pool; this is not 'alignment' in the paper's sense (no configuration is tuned to the victim), yet it could make the original attack succeed where the paper's HiDDeN-only pool fails. This is the key unaddressed alternative explanation for the headline result.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies transfer-based no-box evasion attacks against learning-based image watermarking. It argues that a prior optimization-based attack (Hu et al., 2024) relies on impractical assumptions: the attacker must align surrogate and victim architectures and training configurations, and must train many surrogate models. The authors propose an optimization-free transfer (OFT) attack that uses one surrogate and simply adds the watermark of the flipped decoded secret. They report that relaxing the alignment assumption causes Hu et al.'s attack to fail (evasion rate at most 21.1%), and that in the aligned setting OFT matches or outperforms Hu et al.'s attack in 11 of 12 configurations under the same l_infinity budget. The paper releases code for all experiments.","tokens_in":18938,"tokens_out":7613,"duration_ms":67851,"significance":"If the findings hold, the paper provides a useful reality check on the practicality of transfer-based no-box attacks and introduces a cheap, effective baseline for the aligned setting. Clear strengths are the code release and the systematic variation of surrogate counts and configurations. However, the headline negative result is tested only with a restricted surrogate pool, the aligned comparison uses retuned target models, and no statistical tests are reported despite acknowledged checkpoint variance. The significance is therefore conditional on addressing these gaps.","major_comments":[{"comment":"The principal negative claim—that relaxing the alignment assumption yields evasion rates of at most 21.1%—is supported only by experiments in which the surrogate pool consists exclusively of HiDDeN models. The paper itself notes in §6.2 that Hu et al.'s cross-method evaluation included both HiDDeN and StegaStamp surrogates. Because a no-box attacker who does not know the victim method can still assemble a diverse public surrogate pool, the current experiment tests a strictly weaker version of the original attack. The authors should either replicate Hu et al.'s multi-architecture pool (or a comparable diverse pool) or explicitly restrict the conclusion to HiDDeN-only surrogates. As written, the abstract's 'at most 21.1%' claim is not supported for the original attack as proposed.","section":"§6.2"},{"comment":"The comparison between Hu et al.'s attack and OFT in §6.3 is performed on target models that were re-trained with hyperparameters tuned by the authors. Footnote 5 reports a grid search over encoder and decoder loss weights because the authors found that the hyperparameters from Hu et al. do not reach a reasonable bit accuracy for CNN architectures. This retuning changes the target models relative to those in Hu et al.'s original evaluation, and the paper does not demonstrate that the retuning is neutral between the two attacks. The authors should evaluate on the original target checkpoints as well, or show that the relative ranking is unchanged under both sets of hyperparameters.","section":"§5.1.1"},{"comment":"The paper acknowledges in §7.1 that 'the variance over different checkpoints is not ignorable,' yet Figures 3 and 4 report only means and ranges, and no significance tests are provided for the central '11 out of 12' claim. Given the explicit statement about non-negligible variance, the claim that OFT is superior or comparable in 11 of 12 configurations could be within checkpoint noise. The authors should report confidence intervals or perform paired statistical tests across checkpoints for each configuration, so that the reader can assess the robustness of the headline comparison.","section":"§7.1"}],"minor_comments":[{"comment":"The specific '21.1%' maximum evasion rate for unaligned configurations does not appear in the main text or in a table; please report the exact maximum in §6.2 or in a table so the abstract's number can be verified.","section":"Abstract and §6.2"},{"comment":"The paper compares against 'the best configuration (k = 50)' of Hu et al., but Appendix C.3 notes that k = 40 can be worse than k = 30 (e.g., for ℓ = 64 CNN on MidJourney). The authors should confirm that k = 50 is indeed the best per configuration, or use the per-configuration maximum over k.","section":"§6.3"},{"comment":"There are several typos and minor presentation issues: 'wheras' in Table 2 caption, 'MidJouruney' in Appendix A.1, 'the attack has no knowledge' in the Introduction should be 'the attacker,' and the use of 'BA' for both bit-wise accuracy and bit accuracy in equations should be made consistent.","section":"Throughout"},{"comment":"The reference to 'Figure 6 of their draft 1' is not a stable citation; please cite the arXiv version of Hu et al. with its identifier.","section":"§3.3"},{"comment":"The discussion of why OFT cannot be applied to generator-embedded watermarks is very brief; one concrete example (e.g., Stable Signature) would help the reader understand the scope of the proposed attack.","section":"§7.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a useful empirical critique of a recent attack and proposes a cheap, well-motivated baseline. The main risks are that the cross-method negative result uses only a HiDDeN surrogate pool, whereas the original attack used a diverse pool, and that the aligned comparison retunes target models. Both are fixable with additional experiments and tighter claims. I would also encourage the authors to add basic statistical comparisons, as the paper itself acknowledges large checkpoint variance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a useful empirical paper that does one thing well and one thing incompletely. What's new: it shows that Hu et al.'s transfer attack against image watermarks collapses when the surrogate models are not aligned to the victim, and it offers a trivial baseline—OFT—that matches or beats the optimized attack in most aligned configurations. Code is released, experiments are clearly described, and the paper is honest about its limitations. The OFT result is genuinely interesting: one forward pass through a surrogate encoder, flipping the decoded message, is enough to evade under the same ℓ∞ budget.\n\nThe soft spots are real. The biggest one is exactly what the stress-test note flags: Section 6.2 tests Hu et al.'s attack with only HiDDeN surrogates, while Hu et al.'s own cross-method experiment used a pool containing both HiDDeN and StegaStamp surrogates. A no-box attacker who includes all public watermarking families is not 'aligned' in the paper's sense, but that is the configuration where Hu et al.'s attack was shown to work across methods. By omitting that condition, the paper's headline claim—that relaxing alignment kills the attack—is not actually tested against the original attack as designed. This is a load-bearing omission, and it should be fixed before the negative result is stated broadly.\n\nTwo smaller issues: in Section 5.1.1 the authors re-tune target CNN hyperparameters for ℓ=64 because the released ones didn't reach reasonable accuracy. That's defensible, but it makes the comparison against Hu et al. less clean for that configuration. And there are no significance tests; the authors themselves say the variance over checkpoints is non-ignorable. The plots show means and ranges, which helps, but formal tests would sharpen the 11/12 claim.\n\nNone of this sinks the paper. The OFT baseline is useful regardless, and the empirical finding that optimization gives only limited advantage in aligned settings is a solid, reproducible result. The paper deserves peer review; with the multi-architecture surrogate pool experiment added, it would be a stronger contribution.\n\nI'd bring it to a reading group for the discussion of what 'no-box' realistically means, but I wouldn't rely on the central negative claim in its current form.","headline":"A useful empirical check on a published attack with a cheap baseline, but the headline negative result is tested against a narrower version of the original attack than the paper implies.","tokens_in":19445,"tokens_out":2484,"would_cite":true,"duration_ms":23190,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that optimization-based transfer attacks on image watermarks succeed only under unrealistic assumptions, and that a one-surrogate, optimization-free attack is comparable or better in 11 of 12 configurations.","keywords":["image watermarking","no-box attack","transfer attack","evasion rate","surrogate model","optimization-free attack","HiDDeN","diffusion purification"],"falsifier":"Run the original transfer attack from [20] with the mixed surrogate pool it used (HiDDeN and StegaStamp together) against unaligned targets such as RivaGAN or MBRS under the same $\\ell_\\infty = 0.25$ budget; an evasion rate above 21.1% would contradict the paper's ceiling for relaxed assumptions.","tokens_in":18476,"feed_emoji":"🖼️","tokens_out":12977,"duration_ms":111014,"temperature":0.7,"pith_summary":"This paper argues that optimization-based transfer attacks on image watermarks—attacks that train surrogate watermarking models and optimize a perturbation—look effective only when the surrogate models closely match the victim model's architecture and training setup and when the attacker can afford many surrogates. Once those assumptions are relaxed to a realistic no-box setting, the paper measures a maximum evasion rate of 21.1%. It then proposes Optimization-Free Transfer (OFT), which decodes the watermark with one surrogate, flips the decoded bits, re-encodes that flipped message onto the image, and uses the difference as the perturbation. Under an $\\ell_\\infty$ budget of 0.25 with configurations mostly aligned, OFT is better or comparable to the optimization-based attack in 11 of 12 configurations while being thousands of times faster. If correct, this shifts the practical question from how to optimize transfer attacks to whether optimization has any real advantage over a simple baseline in the no-box setting.","feed_headline":"No-box watermark evasion caps at 21.1% without alignment","feed_subtitle":"Relaxing two assumptions caps evasion at 21.1%; a one-surrogate attack matches the costly baseline in 11 of 12.","key_machinery":"The central object is the Optimization-Free Transfer (OFT) attack, together with the transfer assumption it tests. OFT takes a watermarked image $x_{wm}$, decodes it with a surrogate decoder to get $s_i$, flips all bits to $\\bar{s}_i$, and re-encodes that flipped message with the same surrogate encoder: $\\varepsilon_i = \\mathrm{Enc}_i(\\bar{s}_i, x_{wm}) - x_{wm}$. Perturbations from multiple surrogates are aggregated by mean or median and clamped to the $\\ell_\\infty$ budget. This isolates the paper's core comparison: optimization is removed entirely, so any remaining transfer must come from surrogate–victim alignment rather than from the optimization procedure.","core_discovery":"The central claim is that the success of prior optimization-based transfer attacks depends on two impractical assumptions: alignment between the surrogate and victim watermarking configurations, and large compute for training many surrogates. The paper reports that when the victim method differs from the surrogate method, evasion drops to at most 21.1%, and the only successful configuration is the control where target and surrogate use the same watermarking method. When alignment mostly holds, a single-surrogate, optimization-free attack the authors call OFT—flip the decoded secret, re-encode it as a perturbation, and add it to the image—matches or beats the 50-surrogate optimized attack in 11 of 12 configurations under $\\ell_\\infty \\le 0.25$, losing only narrowly in one 20-bit ResNet setting. OFT also runs in under one second on 1,000 images versus roughly 2,500 seconds for the 50-surrogate optimized attack, and compares favorably to diffusion-purification regeneration baselines.","pith_inferences":["The paper does not test a mixed-architecture surrogate pool; rerunning the original attack with HiDDeN and StegaStamp surrogates together could move the 21.1% ceiling.","OFT can be read as a cheap alignment probe: if a single surrogate evades, the surrogate family is close to the victim; if not, expensive optimization is unlikely to recover transfer.","Applying OFT across watermark families beyond HiDDeN, or combining it with a light regeneration step, would test whether the transfer signal is architectural or data-driven."],"forward_implications":["If the paper is right, transfer-based no-box attacks in the literature overstate the practical threat: under unaligned configurations, evasion is capped at 21.1% at $\\ell_\\infty=0.25$.","OFT becomes the relevant cheap baseline: one surrogate, sub-second runtime, and it beats or ties the 50-surrogate optimization attack in 11 of 12 aligned configurations.","Relaxing compute alone (using OFT) preserves attack success, while relaxing alignment alone kills the optimization attack; alignment, not optimization, is the crucial enabler.","Watermarking deployments that hide their architecture and training configuration gain substantial practical protection against transfer-based evasion."],"supporting_citations":[{"why":"The optimization-based transfer attack whose alignment and compute assumptions the paper relaxes and compares against.","marker":"[20]"},{"why":"Diffusion-purification regeneration baseline used for evasion, runtime, and image-quality comparison.","marker":"[38]"},{"why":"HiDDeN, the watermarking method used for surrogate models and for the aligned target configurations.","marker":"[53]"},{"why":"MBRS, an unaligned target watermarking method in the cross-method transfer experiments.","marker":"[24]"},{"why":"StegaStamp, an unaligned target method and part of the original attack's surrogate pool discussed in Section 6.2.","marker":"[43]"},{"why":"RivaGAN, an unaligned target watermarking method in the transfer experiments.","marker":"[49]"},{"why":"DiffusionDB, one of the two datasets used to train and evaluate target watermarking models.","marker":"[45]"},{"why":"MidJourney dataset, the other dataset used to confirm the aligned-configuration results.","marker":"[44]"}],"fun_headline_variants":["No-box watermark evasion capped at 21.1% without model alignment","Single-model attack beats 50-surrogate transfer in most settings","Watermark attack success drops to 21.1% under realistic assumptions","Optimization-free OFT rivals expensive transfer attacks on watermarks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The negative result assumes that testing the transfer attack with only HiDDeN surrogate models is enough to judge the attack, even though the original attack's cross-method experiment mixed HiDDeN and StegaStamp surrogates.","fun_headline_variants_meta":{"raw":{"variants":["No-box watermark evasion capped at 21.1% without model alignment","Single-model attack beats 50-surrogate transfer in most settings","Watermark attack success drops to 21.1% under realistic assumptions","Optimization-free OFT rivals expensive transfer attacks on watermarks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000588,"raw_usage":{"total_tokens":2796,"prompt_tokens":1019,"completion_tokens":1777,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":635,"completion_tokens_details":{"reasoning_tokens":1702}},"tokens_in":635,"tokens_out":1777,"duration_ms":13383,"temperature":1.0,"reasoning_tokens":1702,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:18:04.275886+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the original transfer attack from [20] with the mixed surrogate pool it used (HiDDeN and StegaStamp together) against unaligned targets such as RivaGAN or MBRS under the same $\\ell_\\infty = 0.25$ budget; an evasion rate above 21.1% would contradict the paper's ceiling for relaxed assumptions.","supporting_citations":[{"cited_title":"Mbrs: Enhancing robustness of dnn-based watermarking by mini-batch of real and simulated jpeg compression","cited_arxiv_id":null,"evidence_quote":"MBRS, an unaligned target watermarking method in the cross-method transfer experiments."},{"cited_title":"Stegastamp: Invisible hyperlinks in physical photographs","cited_arxiv_id":null,"evidence_quote":"StegaStamp, an unaligned target method and part of the original attack's surrogate pool discussed in Section 6.2."},{"cited_title":"Midjourney user prompts & generated images (250k), 2022","cited_arxiv_id":null,"evidence_quote":"MidJourney dataset, the other dataset used to confirm the aligned-configuration results."}],"review_version":1}