{"id":"54071c94-175b-41f8-b0fb-4d20ccb1cb6b","arxiv_id":"2412.15924","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"Watertox combines a two-stage FGSM with a four-model ensemble vote to generate adversarial images that reportedly reduce unseen-model accuracy by up to 98.8%.","lead":"Watertox is a simple adversarial attack that perturbs images in two stages using an ensemble of four models, claiming to knock down accuracy on models it has never seen. The paper argues that even basic, cheap perturbations can transfer across modern vision architectures, which matters for building or breaking visual security systems like CAPTCHAs.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Budget mismatch: Watertox's two-stage update permits per-pixel perturbations up to ε1+ε2=0.5, while the NI-FGSM baseline is restricted to ε=0.1; the claimed advantage (70.6%→16.0%, ASR>95%) may be an artifact of unequal perturbation budgets, not of the ensemble/voting mechanism.","rationale":"The reader's verdict is CONDITIONAL, and my independent reading reaches the same conclusion: the central empirical claim is not adequately supported as written. The reader identified the statistical fragility of the 1,000-image sample (one image per class, no seed, no repeated trials) as the weakest assumption; that is a real reproducibility concern and remains secondary. I see a more load-bearing problem before statistics even enter: the perturbation budget used by Watertox, as defined in §3.2.2, is up to 0.5 in L∞, while the NI-FGSM comparator is explicitly run at ε=0.1. The paper asserts in §4.4.2 that Watertox's modifications were 'strictly bounded' by that 0.1 regime, which contradicts the method's own additive two-stage formulation and its stated bound ∥x_adv2−x∥∞ ≤ ε1+ε2 in §3.3.1. No clipping rule, renormalization, or alternative interpretation is documented, and the table that would show the comparison is missing ('Table ??'). Because the headline accuracy drops and the 'above 95%' ASR claims depend on this comparison, the central claim is underdetermined: a large-budget FGSM-type attack would be expected to degrade models substantially even without any novel ensemble-voting contribution. The visual-fidelity portion of the claim is similarly unbacked, since no quantitative quality metric is reported and a 0.5 L∞ perturbation on natural images is visually large. I am not claiming the method is fraudulent or that the results are impossible; rather, the evidence as presented cannot distinguish a genuine cross-model transfer mechanism from a simple budget-effect. The concrete test I propose—measure the actual L∞ budget and rerun NI-FGSM at the same budget—would settle this concern. If the margin persists under equal budgets, the central claim gains real support; if not, the paper needs substantial revision. This does not change the reader's CONDITIONAL verdict, so I mark the verdict as UNCHANGED, with agreement on the overall assessment but a different primary weak point.","tokens_in":10086,"tokens_out":5091,"duration_ms":47356,"concrete_test":"Recompute the actual per-image L∞ norm of x_adv2−x on the same 1,000-image subset using Algorithm §3.2.2, and then rerun NI-FGSM with the same total budget (ε=0.5, matched number of iterations) and the same surrogate ensemble; additionally report PSNR/SSIM for Watertox at the measured budget. If Watertox's accuracy reductions and ASR margins over NI-FGSM collapse under equal budgets, or if the perturbation quality metrics show severe distortion, the central comparative claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The decisive weakness is an internal inconsistency between the method definition and the comparative protocol. In §3.2.2 the adversarial image is x_adv2 = x + 0.1·sign(∇J_total) + 0.4·sign(losshw). Because losshw is a masked gradient whose sign is nonzero on pixels exceeding the threshold, the per-pixel infinity-norm bound asserted in §3.3.1 is ∥x_adv2−x∥∞ ≤ ε1+ε2 = 0.5, not 0.1. The paper's own threshold analysis says roughly 42.6% of pixels can receive the strong perturbation, so a large fraction of the image is modified at magnitude 0.5. Yet §4.4.2 states that the comparison 'ensured Watertox's modifications remained strictly bounded by NI-FGSM's parameter regime (ε=0.1)' and then compares against NI-FGSM with ε=0.1 and 3 iterations. These statements cannot both hold without an undocumented clipping or budget cap. If Watertox is actually capped at ε=0.1, the algorithm described in §3.2.2 is not what was evaluated; if uncapped, the NI-FGSM comparison is invalid because the baseline is given a five-times-smaller perturbation budget. Since the headline numbers ('70.6% to 16.0%', 'up to 98.8% reduction', ASR>95%) are produced under this budget, they may reflect perturbation magnitude rather than the claimed architectural-diversity and voting contributions. The visual-fidelity claim is also unsupported: no PSNR/SSIM or human-study quantitative metrics are provided, and a 0.5 L∞ perturbation on ImageNet is far beyond typical attack budgets. The missing Table ?? for the NI-FGSM comparison and the unreported t-test details prevent the reader from checking whether any budget-matching actually occurred. This is a correctness risk, not merely a stylistic disagreement with prior literature.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes Watertox, a two-stage FGSM-based adversarial attack that first applies a uniform ε1=0.1 perturbation derived from an ensemble-weighted gradient and then selectively adds ε2=0.4 perturbations to pixels whose gradient magnitude exceeds a threshold. The authors claim strong accuracy reductions on a 1,000-image ImageNet subset for seen architectures (Table 1) and zero-shot architectures (Table 2), and claim superiority over NI-FGSM under a supposedly equal perturbation budget (§4.4). The paper also presents 'theoretical guarantees' for visual quality and cross-model transferability (§3.3).","tokens_in":10526,"tokens_out":6485,"duration_ms":52093,"significance":"If the empirical claims survive scrutiny, Watertox would be a remarkably simple and transferable attack with practical relevance for CAPTCHA and security applications. The paper deserves credit for evaluating against external ImageNet classes and previously unseen architectures, and for reporting raw accuracy tables. However, the significance is currently conditional: the comparison to NI-FGSM rests on an internally inconsistent budget claim and a missing table, the theoretical section is largely asserted rather than proven, and the point-estimate results lack statistical safeguards. These issues must be resolved before the contribution can be assessed.","major_comments":[{"comment":"The method as defined permits per-pixel perturbations up to ε1+ε2=0.5: x_adv2 = x_adv1 + ε2·sign(losshw), so ∥x_adv2−x∥∞ ≤ 0.5. Yet §4.4.2 states that Watertox's modifications were 'strictly bounded by NI-FGSM's parameter regime (ε=0.1)' and compares against NI-FGSM with ε=0.1. These statements are inconsistent unless an undocumented clipping was applied, and if no clipping was applied the comparison is unfair because the baseline is given a five-times-smaller budget. The headline ASR advantage may therefore reflect perturbation magnitude rather than the proposed voting/ensemble mechanism.","section":"§3.2.2, §3.3.1, §4.4.2"},{"comment":"The comparative experiments against NI-FGSM are presented in a missing table ('Table ??'), so the core comparative evidence cannot be checked. Moreover, the text in §4.4.3 is not consistent with Table 3: it claims a 'consistent 98.71% success rate' across targets, whereas Table 3 reports values ranging from 0.9567 to 0.9898, and it reports a 97.5% disruption rate against ResNet152 that conflicts with the 0.9567 ASR in that table. The authors should supply the table and reconcile all numbers.","section":"§4.4.2, §4.4.3, Table 3"},{"comment":"The 'rigorous mathematical guarantees' are not actually derived. The invariant |{i: |loss_i| > avg(|loss|)}| < n/2 is assumed without proof; the 42.6% figure follows from an unjustified assumption that gradient losses are i.i.d. standard normal; and the universal-effectiveness and transferability statements in §3.3.2 are restatements of desired properties rather than theorems with explicit hypotheses and proofs. If these results are meant to be informal motivation, the section should be reframed; if they are meant as guarantees, complete proofs and conditions are needed.","section":"§3.3.1, §3.3.2"},{"comment":"The evaluation uses one randomly selected image per ImageNet class with no stated seed, no repeated trials, and no confidence intervals. The headline accuracy reductions (70.6%→16.0% and up to 98.8%) are point estimates from a single 1,000-image sample and could change substantially under resampling. The paper should report bootstrap confidence intervals or results over multiple random subsets.","section":"§4.1, Tables 1 and 2"},{"comment":"Several components of the method are underspecified, which prevents reproduction: the threshold τ in losshw is never given a numerical value or explicit formula (the invariant in §3.3.1 suggests τ=avg(|loss|), but this is not stated), and the ensemble weights w_j, the regularization term R(x), and λ in J_total are not specified. Please state the exact configuration used in all experiments.","section":"§3.2.1, §3.2.2, §4.1"},{"comment":"The claim that Watertox 'introduces notably less visual distortion' and preserves visual fidelity is not supported by any quantitative metric (e.g., PSNR, SSIM, or a human study), despite the method allowing L∞ perturbations of magnitude 0.5 on many pixels. Please add quantitative perceptual-quality measurements or temper the claim.","section":"§4.2, §4.4.3, Figure 4b"}],"minor_comments":[{"comment":"The text refers to 'Table ??' for the NI-FGSM comparison; this placeholder should be replaced with an actual table.","section":"§4.4.2"},{"comment":"The paper refers to 'ConvNeXT-tiny' in §4.4.2 but 'ConvNeXt-tiny' elsewhere; please standardize the spelling.","section":"§4.4.2"},{"comment":"The abstract and introduction motivate the method for CAPTCHA generation, but no CAPTCHA-specific experiments or human readability evaluation are presented; please clarify the scope.","section":"Abstract, §1, §5.4"},{"comment":"The 42.6% percentage appears to be P(|Z|>sqrt(2/π)) for a standard normal Z, not a property of real gradient distributions; please clarify the intended interpretation.","section":"§3.3.1"},{"comment":"No code or experiment reproduction details (e.g., random seed, software versions) are provided; consider adding an availability statement.","section":"§4.1"}],"recommendation":"major_revision","confidential_remarks":"The central issue is the budget mismatch between Watertox and NI-FGSM; if this cannot be resolved by re-analysis, the authors should remove the comparative claim and present only the absolute accuracy tables. The missing Table ?? and the inconsistencies in §4.4.3 suggest the comparative section was not finalized; the editor may wish to require the authors to supply the full comparison before re-review. The theoretical section could be shortened to a heuristic discussion without affecting the empirical contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is a simple two-pass FGSM plus threshold mask and ensemble sign vote. The ingredients are all known — FGSM, ensemble gradient averaging, sparse thresholding — and the paper cites the key FGSM and transferability works, though it misses Liu et al. ICLR 2017 and the broader ensemble-attack line. What is genuinely nice: the method is easy to state, the zero-shot evaluation uses genuinely unseen architectures (ResNet, ShuffleNet, GoogLeNet, MobileNet), and the raw accuracy tables, if reproducible, would show a strong effect. The ablation table is a useful sanity check that no single surrogate is load-bearing. That is real evidence, but it is not enough.\n\nThe soft spots are large. The stress-test note is right: Section 3.2.2 defines x_adv2 = x_adv1 + 0.4 sign(losshw), and Section 3.3.1 states the L∞ bound is ε1+ε2 = 0.5. Section 4.4.2 then says the comparison 'ensured Watertox's modifications remained strictly bounded by NI-FGSM's parameter regime (ε=0.1)' and compares against NI-FGSM with ε=0.1. These cannot both be true without an undocumented clipping step. If Watertox is actually clipped to 0.1, the evaluated algorithm is not the method described; if it is not clipped, the baseline is given a five-times-smaller budget. Either way the headline numbers (70.6→16.0, up to 98.8% zero-shot) are not interpretable as superiority over NI-FGSM. The missing Table ?? for that comparison removes the possibility of checking what was actually run. The t-test and σ values are reported without the underlying sample sizes or whether the 1,000-image subset was fixed across methods; no seed, no repeated trials, no confidence intervals in Tables 1 and 2. For a single random image per class, the top-1 accuracies could swing by several points. The theoretical guarantees in Section 3.3 are asserted, not proven: the 42.6% figure is just P(|Z| > E|Z|) for a standard normal, not a property of the attack, and the 'guarantees' restate desired behavior. The visual-fidelity claim is also unsupported — no PSNR/SSIM or human study.\n\nIs there a salvageable paper? Maybe. With matched budgets, a code release, seeds and error bars, and the theory removed or replaced by an honest statement that this is an empirical recipe, a short workshop-style paper could be a fine baseline. As submitted, the central comparative claim doesn't hold together, and the missing table means the authors didn't give reviewers the data to check it.\n\nI would not send this to full review as is. If it comes back substantially revised with code and a fair baseline, I'd take another look.\n\nBest.","headline":"Watertox is a clearly written but methodologically flawed two-stage FGSM attack; the headline comparison against NI-FGSM rests on a budget mismatch the paper itself documents, and the missing table and lack of error bars prevent checking the central claim.","tokens_in":11092,"tokens_out":2996,"would_cite":false,"duration_ms":25861,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Watertox, a two-stage FGSM attack with an ensemble voting mechanism, claims to cut a strong ImageNet model from 70.6% to 16.0% and unseen-model accuracy by up to 98.8%.","keywords":["adversarial attack","transferability","zero-shot attack","fast gradient sign method","model ensemble","CAPTCHA","ImageNet","universal adversarial perturbation"],"falsifier":"Repeat the Watertox attack on ten independently drawn 1,000-image subsets, one per ImageNet class, and report the average and spread of post-attack accuracies for ConvNeXt-large and MobileNet V3 Large; if the 16.0% and 1.2% figures are not reproduced within a few percentage points, or if the zero-shot reduction to near 1% disappears, the central claim fails.","tokens_in":9885,"feed_emoji":"🎯","tokens_out":8670,"duration_ms":64663,"temperature":0.7,"pith_summary":"The paper claims that a deliberately simple adversarial attack—two Fast Gradient Sign Method steps, one uniform at $\\epsilon_1=0.1$ and one targeted at $\\epsilon_2=0.4$—can transfer across very different neural network architectures without any tuning for the target. The attack's transferability comes from an ensemble of four architectures (VGG, DenseNet, AlexNet, and ConvNeXt) whose gradient signs are combined by majority vote. On ImageNet, the authors report that the best-performing model drops from 70.6% to 16.0% accuracy, and that zero-shot attacks on six unseen architectures reduce accuracy by up to 98.8%. If these numbers hold, a cheap and universal perturbation recipe exists that could be used for CAPTCHA generation and would expose shared vulnerabilities in current vision models.","feed_headline":"Two-stage FGSM attack cuts top model accuracy to 16 percent","feed_subtitle":"Two epsilon values plus a four-model voting ensemble transfer attacks to architectures the system has never seen.","key_machinery":"The load-bearing object is the two-stage FGSM perturbation $$$x^{{\\mathrm{adv}}$}_2 = x + \\epsilon_1\\,\\mathrm{sign}(\\nabla_x J_{\\mathrm{total}}(x,y)) + \\epsilon_2\\,\\mathrm{sign}(\\mathrm{loss}_{hw}),$$ where $\\mathrm{loss}_{hw}$ is the ensemble gradient masked by a threshold $\\tau$ so that only high-gradient pixels receive the second push. The direction of the ensemble gradient is a weighted sign vote $\\mathrm{sign}(\\sum_j w_j\\,\\mathrm{sign}(\\nabla_j))$ over VGG, DenseNet, AlexNet, and ConvNeXt. This combination—a small uniform push for baseline disruption, a larger push confined to high-gradient regions, and an ensemble sign vote for transferability—carries the argument. The paper adds an $\\ell_\\infty$ bound $\\|x^{\\mathrm{adv}}_2-x\\|_\\infty \\le \\epsilon_1+\\epsilon_2$ to support the visual-fidelity claim and a threshold invariant stating that fewer than half the pixels receive the large perturbation.","core_discovery":"Watertox's central claim is that robust cross-model adversarial examples can be generated by a two-stage FGSM: the first stage applies a uniform perturbation with $\\epsilon_1=0.1$, and the second adds $\\epsilon_2=0.4$ only where the ensemble-averaged gradient magnitude exceeds a threshold. The final perturbation direction is decided by a sign vote across VGG, DenseNet, AlexNet, and ConvNeXt, so no target-specific information is used. Evaluated on a 1,000-image ImageNet subset, this recipe reduces every base model's accuracy, with ConvNeXt-large falling from 70.6% to 16.0%, and it also cuts zero-shot accuracy on six unseen architectures, with MobileNet V3 Large dropping from 63.7% to 1.2%. The paper presents these results as evidence that architectural diversity plus gradient-sign voting yields universal, architecture-agnostic adversarial perturbations.","pith_inferences":["Editorial extension: if the reported reductions survive repeated sampling, the vulnerability is less about a specific architecture and more about the common gradient-sign structure of image classifiers; a defense would need to break that sign alignment rather than merely harden one model.","Editorial extension: a natural test the paper does not run is the same $\\epsilon_1=0.1$, $\\epsilon_2=0.4$ recipe against adversarially trained models, which would show whether the perturbation is truly universal or only effective on undefended baselines.","Editorial extension: the ablation data suggest the ensemble vote may be a minor contributor—removing any one architecture changes success rates by only a few points—so a single surrogate model plus the two-stage FGSM might already explain most of the effect.","Editorial extension: the paper's one-second-per-image figure on a laptop GPU implies the attack could be applied at scale, e.g., to a whole CAPTCHA dataset, but the authors do not measure end-to-end CAPTCHA breaking or human readability, so those application claims remain untested."],"forward_implications":["The same two-stage perturbation, with no target-specific tuning, should degrade accuracy on other ImageNet-trained classifiers, including architectures not in the ensemble.","A CAPTCHA generator could use Watertox to produce images that humans read but current vision models misclassify, because the bounded perturbations are designed to preserve visual structure.","The reported per-image generation time of about one second on a laptop GPU makes the attack practical to deploy before a target model is even chosen.","The near-identical attack success rates across ablation configurations imply that no single ensemble member is indispensable, so the recipe can be simplified further without losing much transferability.","The consistency of the accuracy drops across architecture families supports the paper's claim that shared gradient-sign directions exist across modern vision models."],"supporting_citations":[{"why":"Defines the Fast Gradient Sign Method that Watertox's two-stage perturbation extends.","marker":"[2]"},{"why":"Establishes universal adversarial perturbations, the theoretical basis for perturbations that transfer across models.","marker":"[8]"},{"why":"Shows adversarial examples transfer across architectures, motivating the zero-shot evaluation.","marker":"[7]"},{"why":"NI-FGSM is the state-of-the-art transferability baseline that Watertox compares against in Section 4.4.","marker":"[18]"},{"why":"Supplies the ImageNet validation set from which the 1,000-image test subset is sampled.","marker":"[9]"},{"why":"Ensemble-based attack construction, the precursor to the paper's multi-model voting mechanism.","marker":"[6]"}],"fun_headline_variants":["Universal attack uses two-stage FGSM plus voting to drop accuracy to 16%","Architectural diversity plus gradient voting yields zero-shot adversarial attacks","Two epsilon values and a four-model vote transfer attacks to unseen models","Watertox attack: 98.8% reduction in zero-shot accuracy on unseen models","Simple two-stage FGSM with ensemble vote drops top model to 16% accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline numbers come from a single sample of 1,000 ImageNet images—one per class, chosen at random with no seed, no repeated trials, and no confidence intervals—so the reported accuracy drops could shift substantially if a different sample were used.","fun_headline_variants_meta":{"raw":{"variants":["Universal attack uses two-stage FGSM plus voting to drop accuracy to 16%","Architectural diversity plus gradient voting yields zero-shot adversarial attacks","Two epsilon values and a four-model vote transfer attacks to unseen models","Watertox attack: 98.8% reduction in zero-shot accuracy on unseen models","Simple two-stage FGSM with ensemble vote drops top model to 16% accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000694,"raw_usage":{"total_tokens":3107,"prompt_tokens":883,"completion_tokens":2224,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":499,"completion_tokens_details":{"reasoning_tokens":2122}},"tokens_in":499,"tokens_out":2224,"duration_ms":15361,"temperature":1.0,"reasoning_tokens":2122,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:57:30.195638+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Repeat the Watertox attack on ten independently drawn 1,000-image subsets, one per ImageNet class, and report the average and spread of post-attack accuracies for ConvNeXt-large and MobileNet V3 Large; if the 16.0% and 1.2% figures are not reproduced within a few percentage points, or if the zero-shot reduction to near 1% disappears, the central claim fails.","supporting_citations":[{"cited_title":"ImageNet Large Scale Visual Recognition Challenge,","cited_arxiv_id":null,"evidence_quote":"Supplies the ImageNet validation set from which the 1,000-image test subset is sampled."},{"cited_title":"Universal adversarial perturbations,","cited_arxiv_id":null,"evidence_quote":"Establishes universal adversarial perturbations, the theoretical basis for perturbations that transfer across models."},{"cited_title":"Towards building more robust adversarial examples with ensemble-based methods,","cited_arxiv_id":null,"evidence_quote":"Ensemble-based attack construction, the precursor to the paper's multi-model voting mechanism."}],"review_version":1}