{"id":"e546c2e4-a0ed-433f-bef8-d8d351d015bc","arxiv_id":"2502.09352","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"Extends TRADES to a Wasserstein distributional threat with an efficient budgeted attack, and fine-tunes pre-trained robust models to improve a self-defined distributional robustness metric.","lead":"The paper proposes a fine-tuning method that trains deep networks to resist distributional (Wasserstein) adversarial attacks while preserving pointwise robustness, and tests it on five top CIFAR-10 models. It reports consistent gains against its own distributional attack, but the evaluation is self-referential and lacks independent verification.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (5.5) inverts the per-batch budget scaling, so W-PGD-Budget-20 can leave the δ=8/255 W2 ball; the reported W2 adversarial accuracies do not measure the claimed threat.","rationale":"The reader's weakest assumption targets the faithfulness of W-PGD-Budget as an approximation of the true worst-case distributional attack. I agree that this is the load-bearing point, but the paper contains a more specific and checkable internal difficulty: Eq. (5.5) appears to invert the per-batch budget. Deriving the budget from Theorem 4.1 gives δ_B=δΥ/Υ_B, whereas the paper states the reciprocal. If taken literally, the evaluation attack is not constrained to the claimed W2 ball, so the experimental evidence for improved distributional robustness is not valid as reported. This is an internal inconsistency, not a disagreement with external consensus. I keep the reader's CONDITIONAL verdict because the flaw may be a typo that code release or a corrected derivation could resolve; a REJECT would be premature without checking the actual implementation.","tokens_in":12998,"tokens_out":10011,"duration_ms":104245,"concrete_test":"Run W-PGD-Budget-20 with the released code on the fine-tuned Zhang et al. (2019) model and compute the empirical W2 distance between the original and attacked test distributions under metric (4.1). If the average squared l∞ displacement exceeds (8/255)^2, or equivalently if the empirical W2 distance exceeds 8/255, then Eq. (5.5) violates the stated budget. As a complementary check, replace δ_B=δΥ_B/Υ with δ_B=δΥ/Υ_B and re-run Table 5; if the reported W2 accuracies shift materially, the published improvements are artifacts of the invalid budget scaling.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Under Theorem 4.1, the first-order W2 attack pushes each sample by δT(x) with T(x)=Υ^{-1}sgn(∇J)||∇J||_1, and the W2 constraint holds because E||δT||_∞^2 = δ^2, using Υ^2 = E||∇J||_1^2. For a batch B with Υ_B=(E_{P_B}||∇J||_1^2)^{1/2}, the local displacement has squared l∞ norm δ_B^2||∇J||_1^2/Υ^2. To keep the global coupling inside B_2(P,δ), one needs E_{P_B}[δ_B^2||∇J||_1^2/Υ^2]=δ^2, i.e. δ_B=δΥ/Υ_B. Equation (5.5) instead defines δ_B=δΥ_B/Υ, the reciprocal. As written, a batch whose gradient norm is twice the global average receives twice the global budget, and the global W2 distance of the coupling exceeds δ. W-PGD-Budget-20 is therefore not an admissible attack at the claimed radius. Since Algorithm 1 uses this attack for training and Section 6 uses W-PGD-20 for evaluation, the W2 adversarial accuracies in Tables 2, 4 and 5 are not measurements of robustness within B_2(P,8/255). The central claim may survive if (5.5) is a typo, but the published derivation does not support it.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a Wasserstein distributionally robust training method for deep neural networks, extending TRADES to a distributional threat model in which the adversary perturbs the empirical distribution within a W2 ball using an l∞ ground metric. The training objective (5.1) is motivated by a first-order sensitivity approximation of the inner supremum, and the authors introduce a computationally feasible W-PGD-Budget attack for minibatch training. The main experimental claim is that fine-tuning five adversarially robust CIFAR-10 models from RobustBench with this objective for at most 20 epochs improves their W2 adversarial accuracy under a 20-step W-PGD attack with ReDLR loss, while largely preserving W∞ pointwise accuracy. The paper is honest about limitations, noting that models pre-trained on very large synthetic datasets show smaller gains and some loss of pointwise robustness.","tokens_in":13365,"tokens_out":11924,"duration_ms":123630,"significance":"If the reported results are robust, the contribution is practically valuable: it offers a relatively cheap fine-tuning procedure that can improve distributional robustness of already strong pointwise-robust models, which is a less studied threat model. The use of first-order Wasserstein sensitivity analysis is principled, and the paper builds on a clear line of prior work, including the authors' own W-PGD attack. The manuscript also has explicit strengths: it proposes a concrete algorithm, reports results on several standard RobustBench models, and includes a limitations section with negative results. However, the central experimental evidence currently rests on a single attack family that is also used for training, and the batch-budget scaling in the training attack is not theoretically valid as written. The significance therefore depends on whether these issues can be resolved in revision.","major_comments":[{"comment":"The per-batch budget scaling is the reciprocal of what is needed for an admissible attack. With Υ^2 = E_P ||∇_{x'}J(x,x,y)||_1^2 and Υ_B^2 = E_{P_B}||∇_{x'}J(x,x,y)||_1^2, the first-order displacement of a sample is δ_B ||∇J||_1 / Υ, so the batch's contribution to W_2(P,Q)^2 is δ_B^2 Υ_B^2 / Υ^2. Requiring this to equal δ^2 gives δ_B = δ Υ / Υ_B, not δ_B = δ Υ_B / Υ as printed in Eq. (5.5) and Algorithm 1. With the printed formula, a batch whose gradient norm is twice the global average receives twice the global budget, so the resulting coupling is not in B_2(P,δ), and the statement in §5 that the attack is 'clearly' admissible is false as written. This does not immediately invalidate the W-PGD-20 evaluation in Tables 4 and 5, because that evaluation uses the global attack, but it does break the claimed theoretical grounding of the training attack in Algorithm 1. Please correct the scaling (or explicitly confirm it is a typo and restate the derivation) and discuss whether the reported experiments used the printed formula or the correct one.","section":"§5, Eq. (5.5) and Algorithm 1"},{"comment":"All distributional robustness numbers are computed with the same 20-step W-PGD attack with ReDLR loss that is used in training, and the best epoch is selected on the validation set using the same metric. This creates a risk that the reported improvements, especially the +9.66 percentage-point gain for Chen and Lee (2024), reflect adaptation to this specific attack rather than a genuine improvement in distributional robustness. The paper would be substantially stronger if it included an independent distributional evaluation, for example a different loss, a larger number of steps, or a different optimizer for the inner maximization, as well as error bars over repeated runs. This point should be addressed before the central claim can be accepted.","section":"§6 and Tables 4–5"},{"comment":"Theorem 4.1 assumes the map (x,x',y) ↦ J_θ(x,x',y) is Lipschitz, but the attack loss used in (5.2) and Algorithm 1 is ReDLR, a ratio of logits. For fixed θ, ReDLR can be unbounded when the top logits are nearly tied, and its gradient is not obviously bounded on the compact input domain, so the Lipschitz condition and the uncontrolled o(δ) error term are not verified for the loss actually used. Please either prove or cite a bound for the Lipschitz constant of ReDLR for the considered networks, use a regularized version of the loss, or state explicitly that the first-order sensitivity is applied heuristically.","section":"§4, Theorem 4.1 and §5, Eq. (5.2)"}],"minor_comments":[{"comment":"The column headings and values in Table 2 are confusing and appear inconsistent with Table 1: for Zhang et al. (2019), the W∞ value in Table 2 is 70.61 while Table 1 reports 57.05. Please clarify whether Table 2 reports accuracies under three different attacks on the original models, and align the notation with Table 1.","section":"Table 2"},{"comment":"Each configuration appears to be a single run, and the reported epoch is chosen as the best on the validation set; standard errors over at least three runs, or an explicit statement about the absence of repeated runs, are needed to assess the stability of the improvements and of the W∞ decreases of -3.05 and -2.21.","section":"Tables 3–5"},{"comment":"The initialization of Υ before the first moving-average update is not specified; state how Υ is initialized (for example, from the first 10% sample).","section":"Algorithm 1"},{"comment":"The symbol J_θ is used for both the attack loss (ReDLR) and the training loss (CE plus β times CE of logits), without explicit distinction; define J_θ^{attack} and J_θ^{train} to avoid ambiguity.","section":"§5 and Algorithm 1"},{"comment":"The text says fine-tuning uses the original training dataset (50k images) and mentions 50k images, but the experiments use a 40k training split with 10k held out for validation; please correct this wording in the abstract and Section 6.","section":"Abstract and §6"},{"comment":"There is an architecture inconsistency: Table 1 lists Chen and Lee (2024) as WDR 34-10, whereas Section 6 describes it as WideResNet 28-10; correct the table.","section":"Table 1 and §6"},{"comment":"The global projection pj_δ is not defined beyond its name; provide a reference or pseudocode showing how the projection onto B_2(P,δ) is implemented, since the admissibility of the evaluation attack depends on it.","section":"§5, Eq. (5.3)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript depends heavily on the authors' own prior W-PGD and ReDLR framework, and both the training and the evaluation use that framework. This is not in itself disqualifying, but it makes independent verification important; I would encourage the editor to require an independent distributional attack in the revision. The budget-scaling issue in Eq. (5.5) is likely fixable as a typo, but as published it is a real technical error in the derivation of the training algorithm. No code or computational cost is reported, which also limits the practical-utility claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First: the core evaluation machinery has a concrete error. Eq. (5.5) and Algorithm 1 set the per-batch budget to δ_B = δΥ_B/Υ. From Theorem 4.1, a batch's displacement contributes δ_B^2 Υ_B^2/Υ^2 to the global squared W2 cost; to stay inside B_2(P, δ) you need δ_B = δΥ/Υ_B, the reciprocal. So the attack as specified overshoots the ball for high-variance batches, and the sentence in Section 5 claiming the batch budgets \"would lead to an admissible attack\" is false as written. The W2 accuracies in Tables 2, 4, and 5 are therefore not measurements of robustness at the stated radius. The direction is mildly interesting — an overshooting attack is stronger, so the numbers are conservative lower bounds — but the paper still doesn't measure what it claims, and because the same attack drives training and evaluation, the gains may reflect fit to an attack at the wrong radius. If Eq. (5.5) is a typo and the implementation used the corrected scaling, the central idea probably survives; as published, the empirical claims need revalidation.\n\nCredit where due: the W2 TRADES objective is a clean extension, the W-PGD-Budget speedup (10% sample, moving-average Υ) is a sensible workaround to a real computational problem, and the fine-tuning study is honestly run — they report the failed last-layer-only results, note the W∞ drops on synthetic-data models, and the Limitations section is candid about computational limits. Citations to Bartl et al. and Bai et al. are proper; this is incremental within an established program, which the authors don't conceal.\n\nThe other soft spots are the ones the reader flagged. The train/eval loop uses the same attack family and ReDLR loss, so circularity is real, though there's no AutoAttack equivalent for W2 to lean on. No error bars, and best-epoch-on-validation selection inflates optimism. The uncontrolled o(δ) term in the first-order sensitivity is inherent to this approach and I don't weight it heavily.\n\nBottom line: I'd send this to a serious referee, because the flaw is checkable and fixable and the recipe is useful if it holds. I'd want the formula fixed, an independent attack or at least a second evaluation loss, and error bars before trusting the numbers.","headline":"A sensible W2 fine-tuning recipe undermined by an inverted per-batch budget formula, so the reported W2 accuracies do not measure the claimed threat.","tokens_in":13820,"tokens_out":8251,"would_cite":false,"duration_ms":85442,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","49Q22"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that sensitivity-guided fine-tuning can train pre-trained neural networks to withstand Wasserstein distributional attacks—whole-distribution perturbations within a W2 ball—while keeping their existing pointwise…","keywords":["Wasserstein distributionally robust optimization","adversarial training","distributional adversarial attacks","W-PGD attack","TRADES","fine-tuning deep neural networks","CIFAR-10","sensitivity analysis"],"falsifier":"Evaluate the fine-tuned models under a stronger W2 attack that does not use the 10%-sample sensitivity estimate or the per-batch budget scaling—for instance, a full-dataset W-PGD-20 or an exact solve of the inner W-DRO supremum—and compare the resulting W2 adversarial accuracy with the paper's table. If the gains shrink to zero or reverse sign, the central claim fails.","tokens_in":12813,"feed_emoji":"🛡️","tokens_out":12512,"duration_ms":113560,"temperature":0.7,"pith_summary":"The paper proposes a training objective that extends TRADES—a widely used pointwise adversarial training loss—to Wasserstein distributional threats, where an attacker may move the whole data distribution a bounded distance in the W2 metric instead of perturbing each image separately. It turns the inner worst-case maximization into a computable attack using a first-order sensitivity approximation, and packages this as a 20-epoch fine-tuning routine for networks that are already pointwise-robust. On five pre-trained CIFAR-10 networks, the fine-tuning raised W2 adversarial accuracy in every case, with gains from 1.15 to 9.66 percentage points. For the two networks trained only on the original dataset, pointwise W∞ accuracy also rose; for the three trained on huge external or synthetic datasets, W∞ dipped slightly. If the approximate attack faithfully represents the W2 threat, this gives a practical way to upgrade existing models against distributional attacks without retraining from scratch.","feed_headline":"Fine-tuning lifts W2 attack accuracy 9.7 points on CIFAR-10","feed_subtitle":"Twenty epochs of fine-tuning lift accuracy under distributional attacks while pointwise defense holds nearly steady.","key_machinery":"The machinery is a first-order sensitivity expansion of the worst-case loss, $V(\\delta)=V(0)+\\delta\\Upsilon+o(\\delta)$, where $\\Upsilon = (E_P\\|\\nabla_{x'}J_\\theta(x,x,y)\\|_1^2)^{1/2}$ is the root-mean-square $\\ell^1$ norm of the loss gradient. The expansion replaces the infinite-dimensional maximization over couplings with a single attack step: each image moves to $x+\\delta T(x)$ with $T(x)=\\Upsilon^{-1}\\,\\mathrm{sgn}(\\nabla_{x'}J_\\theta(x,x,y))\\|\\nabla_{x'}J_\\theta(x,x,y)\\|_1$. Since $\\Upsilon$ and the projection onto the Wasserstein ball are global quantities, the paper introduces W-PGD-Budget, which estimates $\\Upsilon$ from a 10% sample with a moving average and scales each batch's budget by the ratio of local to global sensitivity. The ReDLR loss is used in the attack step so that budget is spent on correctly classified images, while cross-entropy is used for parameter updates.","core_discovery":"The central claim is that Wasserstein distributional robustness can be induced by fine-tuning: minimize the clean cross-entropy plus a regularizer penalizing worst-case W2 perturbations, with the worst case computed by sensitivity analysis. On five pre-trained CIFAR-10 networks evaluated under a 20-step W2 attack, the procedure improved W2 adversarial accuracy by 1.15 to 9.66 percentage points while preserving pointwise robustness: the two networks trained only on CIFAR-10 also improved under l∞ attacks, while the three trained on 20–100M extra images lost between 1.43 and 3.05 points in W∞ accuracy. The paper presents this as evidence that distributional robustness is a distinct, trainable axis that can be added to an already strong pointwise model.","pith_inferences":["Because training and evaluation share the same first-order W-PGD-Budget approximation, the honest reading is that the method optimizes for that specific attack family; a strictly stronger W2 attacker could yield smaller gains.","The success of last-layer-only fine-tuning on synthetic-data models, but not on original-data models, suggests that the location of the distributional vulnerability shifts with pretraining data scale—a testable hypothesis for follow-up work.","The same sensitivity expansion extends to general $(l_r, W_p)$ threat models, so the recipe could be adapted to other image metrics or data domains without new theory, though the paper does not test those settings.","A natural next step, which the paper leaves open, is training from scratch with a combined W2 + W∞ objective; the observed W∞ drops on externally pretrained models indicate that such joint training would be needed to get both guarantees without trade-off."],"forward_implications":["Twenty epochs of fine-tuning on the original 50k-image training set can be enough to improve W2 adversarial accuracy on a pre-trained network, with gains between +1.15 and +9.66 percentage points across the five tested configurations.","For networks pretrained only on CIFAR-10, distributional fine-tuning improves both W2 and W∞ adversarial accuracy, so the two robustness goals do not necessarily trade off in this regime.","For networks pretrained on 20–100M external or synthetic images, W2 accuracy still rises but pointwise W∞ accuracy drops slightly, between −1.43 and −3.05 points, so users of such models would face a small trade-off.","Using the current sensitivity estimate rather than a moving average (η = 1 versus η = 0.1) gave better results in all tested settings, and full-network fine-tuning with a small learning rate worked best for models trained on the original dataset."],"supporting_citations":[{"why":"supplies the W-PGD attack, the ReDLR loss, and the sensitivity-based first-order distributional threat that the training method extends.","marker":"Bai et al. (2023)"},{"why":"provides the sensitivity analysis theorem used to derive the first-order approximation V(delta)=V(0)+delta*Upsilon+o(delta).","marker":"Bartl et al. (2021)"},{"why":"is the TRADES objective that the paper extends to distributional threats and one of the five pre-trained networks.","marker":"Zhang et al. (2019)"},{"why":"defines PGD adversarial training, the pointwise baseline that the fine-tuning must preserve.","marker":"Madry et al. (2018)"},{"why":"supplies the unified W-DRO coupling formulation that motivates the distributional training objective.","marker":"Bui et al. (2022)"},{"why":"provides the standardized benchmark and the pre-trained networks used in the evaluation.","marker":"Croce et al. (2021)"},{"why":"gives the convex-duality perspective on Wasserstein DRO that the sensitivity approach avoids.","marker":"Blanchet and Murthy (2019)"},{"why":"represents prior Wasserstein DRO training with stronger smoothness assumptions, which the paper contrasts with its weaker Lipschitz assumption.","marker":"Sinha et al. (2018)"}],"fun_headline_variants":["Fine-tuning boosts W2 robustness without hurting pointwise defense","Add distributional robustness via fine-tuning: up to 9.7 pts on CIFAR-10","W2 attack accuracy jumps up to 9.7 pts after fine-tuning","New method enhances Wasserstein robustness in pre-trained nets","Distributional fine-tuning lifts W2 accuracy, keeps l∞ defense"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the approximate attack used for both training and evaluation faithfully represents the true worst-case distributional threat, so the measured accuracy gains are real defense improvements rather than artifacts of the approximation.","fun_headline_variants_meta":{"raw":{"variants":["Fine-tuning boosts W2 robustness without hurting pointwise defense","Add distributional robustness via fine-tuning: up to 9.7 pts on CIFAR-10","W2 attack accuracy jumps up to 9.7 pts after fine-tuning","New method enhances Wasserstein robustness in pre-trained nets","Distributional fine-tuning lifts W2 accuracy, keeps l∞ defense"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000718,"raw_usage":{"total_tokens":3182,"prompt_tokens":859,"completion_tokens":2323,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":475,"completion_tokens_details":{"reasoning_tokens":2227}},"tokens_in":475,"tokens_out":2323,"duration_ms":16085,"temperature":1.0,"reasoning_tokens":2227,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T21:46:43.633049+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate the fine-tuned models under a stronger W2 attack that does not use the 10%-sample sensitivity estimate or the per-batch budget scaling—for instance, a full-dataset W-PGD-20 or an exact solve of the inner W-DRO supremum—and compare the resulting W2 adversarial accuracy with the paper's table. If the gains shrink to zero or reverse sign, the central claim fails.","supporting_citations":[],"review_version":1}