REVIEW 3 major objections 7 minor 13 references
Wasserstein distributional adversarial training for deep neural networks
T0 review · 3 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read The paper claims that sensitivity-guided fine-tuning can train pre-trained neural networks to withstand Wasserstein distributional attacks—whole-distribution perturbations within a W2 ball—while keeping their existing pointwise…
desk verdict A sensible W2 fine-tuning recipe undermined by an inverted per-batch budget formula, so the reported W2 accuracies do not measure the claimed threat. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is a first-order sensitivity expansion of the worst-case loss, $V(\delta)=V(0)+\delta\Upsilon+o(\delta)$, where $\Upsilon = (E_P\|\nabla_{x'}J_\theta(x,x,y)\|_1^2)^{1/2}$ is the root-mean-square $\ell^1$ norm of the loss gradient. The expansion replaces the infinite-dimensional maximization over couplings with a single attack step: each image moves to $x+\delta T(x)$ with $T(x)=\Upsilon^{-1}\,\mathrm{sgn}(\nabla_{x'}J_\theta(x,x,y))\|\nabla_{x'}J_\theta(x,x,y)\|_1$. Since $\Upsilon$ and the projection onto the Wasserstein ball are global quantities, the paper introduces W-PGD-Budget, which estimates $\Upsilon$ from a 10% sample with a moving average and scales each batch's budget by the ratio of local to global sensitivity. The ReDLR loss is used in the attack step so that budget is spent on correctly classified images, while cross-entropy is used for parameter updates.
What would settle it
Evaluate the fine-tuned models under a stronger W2 attack that does not use the 10%-sample sensitivity estimate or the per-batch budget scaling—for instance, a full-dataset W-PGD-20 or an exact solve of the inner W-DRO supremum—and compare the resulting W2 adversarial accuracy with the paper's table. If the gains shrink to zero or reverse sign, the central claim fails.
Extended reading notes
Core claim
The central claim is that Wasserstein distributional robustness can be induced by fine-tuning: minimize the clean cross-entropy plus a regularizer penalizing worst-case W2 perturbations, with the worst case computed by sensitivity analysis. On five pre-trained CIFAR-10 networks evaluated under a 20-step W2 attack, the procedure improved W2 adversarial accuracy by 1.15 to 9.66 percentage points while preserving pointwise robustness: the two networks trained only on CIFAR-10 also improved under l∞ attacks, while the three trained on 20–100M extra images lost between 1.43 and 3.05 points in W∞ accuracy. The paper presents this as evidence that distributional robustness is a distinct, trainable axis that can be added to an already strong pointwise model.
Load-bearing premise
The load-bearing premise is that the approximate attack used for both training and evaluation faithfully represents the true worst-case distributional threat, so the measured accuracy gains are real defense improvements rather than artifacts of the approximation.
Editorial extensions
If this is right
- Twenty epochs of fine-tuning on the original 50k-image training set can be enough to improve W2 adversarial accuracy on a pre-trained network, with gains between +1.15 and +9.66 percentage points across the five tested configurations.
- For networks pretrained only on CIFAR-10, distributional fine-tuning improves both W2 and W∞ adversarial accuracy, so the two robustness goals do not necessarily trade off in this regime.
- For networks pretrained on 20–100M external or synthetic images, W2 accuracy still rises but pointwise W∞ accuracy drops slightly, between −1.43 and −3.05 points, so users of such models would face a small trade-off.
- Using the current sensitivity estimate rather than a moving average (η = 1 versus η = 0.1) gave better results in all tested settings, and full-network fine-tuning with a small learning rate worked best for models trained on the original dataset.
Reading between the lines
- Because training and evaluation share the same first-order W-PGD-Budget approximation, the honest reading is that the method optimizes for that specific attack family; a strictly stronger W2 attacker could yield smaller gains.
- The success of last-layer-only fine-tuning on synthetic-data models, but not on original-data models, suggests that the location of the distributional vulnerability shifts with pretraining data scale—a testable hypothesis for follow-up work.
- The same sensitivity expansion extends to general $(l_r, W_p)$ threat models, so the recipe could be adapted to other image metrics or data domains without new theory, though the paper does not test those settings.
- A natural next step, which the paper leaves open, is training from scratch with a combined W2 + W∞ objective; the observed W∞ drops on externally pretrained models indicate that such joint training would be needed to get both guarantees without trade-off.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a Wasserstein distributionally robust training method for deep neural networks, extending TRADES to a distributional threat model in which the adversary perturbs the empirical distribution within a W2 ball using an l∞ ground metric. The training objective (5.1) is motivated by a first-order sensitivity approximation of the inner supremum, and the authors introduce a computationally feasible W-PGD-Budget attack for minibatch training. The main experimental claim is that fine-tuning five adversarially robust CIFAR-10 models from RobustBench with this objective for at most 20 epochs improves their W2 adversarial accuracy under a 20-step W-PGD attack with ReDLR loss, while largely preserving W∞ pointwise accuracy. The paper is honest about limitations, noting that models pre-trained on very large synthetic datasets show smaller gains and some loss of pointwise robustness.
Significance. If the reported results are robust, the contribution is practically valuable: it offers a relatively cheap fine-tuning procedure that can improve distributional robustness of already strong pointwise-robust models, which is a less studied threat model. The use of first-order Wasserstein sensitivity analysis is principled, and the paper builds on a clear line of prior work, including the authors' own W-PGD attack. The manuscript also has explicit strengths: it proposes a concrete algorithm, reports results on several standard RobustBench models, and includes a limitations section with negative results. However, the central experimental evidence currently rests on a single attack family that is also used for training, and the batch-budget scaling in the training attack is not theoretically valid as written. The significance therefore depends on whether these issues can be resolved in revision.
major comments (3)
- [§5, Eq. (5.5) and Algorithm 1] The per-batch budget scaling is the reciprocal of what is needed for an admissible attack. With Υ^2 = E_P ||∇_{x'}J(x,x,y)||_1^2 and Υ_B^2 = E_{P_B}||∇_{x'}J(x,x,y)||_1^2, the first-order displacement of a sample is δ_B ||∇J||_1 / Υ, so the batch's contribution to W_2(P,Q)^2 is δ_B^2 Υ_B^2 / Υ^2. Requiring this to equal δ^2 gives δ_B = δ Υ / Υ_B, not δ_B = δ Υ_B / Υ as printed in Eq. (5.5) and Algorithm 1. With the printed formula, a batch whose gradient norm is twice the global average receives twice the global budget, so the resulting coupling is not in B_2(P,δ), and the statement in §5 that the attack is 'clearly' admissible is false as written. This does not immediately invalidate the W-PGD-20 evaluation in Tables 4 and 5, because that evaluation uses the global attack, but it does break the claimed theoretical grounding of the training attack in Algorithm 1. Please correct the scaling (or explicitly confirm it is a typo and restate the derivation) and discuss whether the reported experiments used the printed formula or the correct one.
- [§6 and Tables 4–5] All distributional robustness numbers are computed with the same 20-step W-PGD attack with ReDLR loss that is used in training, and the best epoch is selected on the validation set using the same metric. This creates a risk that the reported improvements, especially the +9.66 percentage-point gain for Chen and Lee (2024), reflect adaptation to this specific attack rather than a genuine improvement in distributional robustness. The paper would be substantially stronger if it included an independent distributional evaluation, for example a different loss, a larger number of steps, or a different optimizer for the inner maximization, as well as error bars over repeated runs. This point should be addressed before the central claim can be accepted.
- [§4, Theorem 4.1 and §5, Eq. (5.2)] Theorem 4.1 assumes the map (x,x',y) ↦ J_θ(x,x',y) is Lipschitz, but the attack loss used in (5.2) and Algorithm 1 is ReDLR, a ratio of logits. For fixed θ, ReDLR can be unbounded when the top logits are nearly tied, and its gradient is not obviously bounded on the compact input domain, so the Lipschitz condition and the uncontrolled o(δ) error term are not verified for the loss actually used. Please either prove or cite a bound for the Lipschitz constant of ReDLR for the considered networks, use a regularized version of the loss, or state explicitly that the first-order sensitivity is applied heuristically.
minor comments (7)
- [Table 2] The column headings and values in Table 2 are confusing and appear inconsistent with Table 1: for Zhang et al. (2019), the W∞ value in Table 2 is 70.61 while Table 1 reports 57.05. Please clarify whether Table 2 reports accuracies under three different attacks on the original models, and align the notation with Table 1.
- [Tables 3–5] Each configuration appears to be a single run, and the reported epoch is chosen as the best on the validation set; standard errors over at least three runs, or an explicit statement about the absence of repeated runs, are needed to assess the stability of the improvements and of the W∞ decreases of -3.05 and -2.21.
- [Algorithm 1] The initialization of Υ before the first moving-average update is not specified; state how Υ is initialized (for example, from the first 10% sample).
- [§5 and Algorithm 1] The symbol J_θ is used for both the attack loss (ReDLR) and the training loss (CE plus β times CE of logits), without explicit distinction; define J_θ^{attack} and J_θ^{train} to avoid ambiguity.
- [Abstract and §6] The text says fine-tuning uses the original training dataset (50k images) and mentions 50k images, but the experiments use a 40k training split with 10k held out for validation; please correct this wording in the abstract and Section 6.
- [Table 1 and §6] There is an architecture inconsistency: Table 1 lists Chen and Lee (2024) as WDR 34-10, whereas Section 6 describes it as WideResNet 28-10; correct the table.
- [§5, Eq. (5.3)] The global projection pj_δ is not defined beyond its name; provide a reference or pseudocode showing how the projection onto B_2(P,δ) is implemented, since the admissibility of the evaluation attack depends on it.
Circularity Check
W2 robustness claims are evaluated with the same ReDLR-based W-PGD attack family used for training, making the headline improvements a fit to the evaluation metric rather than an independent test.
-
fitted input called prediction
[Section 5 (Details of the AA Step, Eq. 5.2 and Algorithm 1) and Section 6 (Evaluation Protocol)]
"For the AA step we apply the W-PGD-20 attack developed in Bai et al. (2023) to supπ∈Π2(P,δ) Eπ[ReDLR(fθ(x), fθ(x′))]. ... The adversarial accuracy reported in this paper is evaluated under 20-step W-PGD attack on the test set with CE loss for the pointwise threat and with ReDLR loss for the distributional threat."
The same ReDLR-based W-PGD attack family generates the training perturbations (Algorithm 1: W-PGD-Budget with ReDLR; Eq. 5.2) and defines the reported W2 adversarial accuracy (Section 6: W-PGD-20 with ReDLR). Algorithm 1 updates θ by minimizing CE at W-PGD-Budget adversarial examples, so the model is explicitly fitted to survive this attack. The reported W2 improvements in Table 5 (up to +9.66 pp) are then measured under a global-budget version of the same attack; they do not test robustness against the true worst-case W2 perturbation. No independent attack or certified bound is used, and the best epoch/hyperparameters are selected on the same W2 validation metric.
-
self citation load bearing
[Section 3 (ReDLR Loss) and Section 5 (Details of the AA Step)]
"In Bai et al. (2023), however, it is pointed out that DLR loss is not appropriate for distributional threat models. The authors instead propose ReDLR loss, a rectified version of DLR loss. ... It has been verified that ReDLR gives a more effective distributional adversarial attack compared to the one using KL divergence or DLR loss, see Bai et al. (2023)."
The central design choice—ReDLR loss and the W-PGD-20 attack—is adopted from Bai et al. (2023), whose author list (Bai, He, Jiang, Oblój) is identical to the present paper. The suitability of ReDLR for distributional threats, and the effectiveness of W-PGD-20, are asserted by self-citation rather than independently re-derived or benchmarked. Because this same attack and loss define the paper's outcome metric, the empirical claim of improved Wasserstein distributional robustness is load-bearing on an unverified self-citation: if ReDLR/W-PGD is a poor surrogate for the W2 worst case, both the training objective and the reported improvement fail in the same way, and no external evidence is available to detect it.
full rationale
The mathematical scaffolding is not itself circular: Theorem 4.1 is stated as adapted from Bartl et al. (2021, Theorem 2.2) and Bai et al. (2023, Theorem 4.1) with proofs, and the W2 analog of TRADES in Eq. (5.1) is a legitimate extension of a published objective. Pointwise W∞ robustness is evaluated with a standard 20-step PGD attack, which provides some independent content. The circularity enters specifically in the distributional-robustness claim: the training attack (W-PGD-Budget, Algorithm 1) and the evaluation attack (W-PGD-20, Section 6) are both members of the same self-cited W-PGD family using the same ReDLR loss, and the reported W2 improvements are selected by the best W2 validation epoch. Thus the headline result is substantially a self-fit rather than a prediction against an independent or certified adversary. Separately, and as a correctness concern rather than a circularity reduction, Eq. (5.5) defines δB = δΥB/Υ, whereas preserving the global W2 budget requires the reciprocal scaling δB = δΥ/ΥB; if this is not a typo, the W-PGD-Budget attack can leave the claimed B2(P,δ) ball, so the reported W2 accuracies may not measure the stated threat at all.
Assumptions & free parameters
free parameters (8)
- beta =
6
- delta =
8/255
- learning rate tau =
0.1 or 0.01
- update rate eta =
0.1 or 1
- noise scale for weight perturbation =
not specified
- sampling fraction =
0.1
- epochs =
20
- batch size =
512
assumptions (4)
- domain assumption First-order sensitivity approximation (Theorem 4.1) is valid for the neural network loss.
- ad hoc to paper Batch-local budget scaling in Eq. (5.5) produces a valid global W2 attack.
- domain assumption ReDLR loss is the appropriate loss for distributional attacks.
- ad hoc to paper Best-epoch-on-validation selection gives an unbiased estimate of test performance.
Cite this review
Pith. "Pith review of Wasserstein distributional adversarial training for deep neural networks." pith.science (2026). https://pith.science/paper/TSRBQIUV
@misc{pith2026250209352,
author = {Pith},
title = {Pith review of: Wasserstein distributional adversarial training for deep neural networks},
year = {2026},
howpublished = {\url{https://pith.science/paper/TSRBQIUV}},
note = {Machine review of arXiv:2502.09352}
}
read the original abstract
Design of adversarial attacks for deep neural networks, as well as methods of adversarial training against them, are subject of intense research. In this paper, we propose methods to train against distributional attack threats, extending the TRADES method used for pointwise attacks. Our approach leverages recent contributions and relies on sensitivity analysis for Wasserstein distributionally robust optimization problems. We introduce an efficient fine-tuning method which can be deployed on a previously trained model. We test our methods on a range of pre-trained models on RobustBench. These experimental results demonstrate the additional training enhances Wasserstein distributional robustness, while maintaining original levels of pointwise robustness, even for already very successful networks. The improvements are less marked for models pre-trained using huge synthetic datasets of 20-100M images. However, remarkably, sometimes our methods are still able to improve their performance even when trained using only the original training dataset (50k images).
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[4]
J. Howard and S. Ruder. Universal language model fine-tuning for text classification. In I. Gurevych and Y. Miyao, editors, Proceedings of the 56th Annual Meeting of the Association for Computa- tional Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers , pages 328–339. Association for Computational Linguistics,
work page 2018
-
[6]
S. Kornblith, J. Shlens, and Q. V. Le. Do better imagenet models transfer better? In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019 , pages 2661–2671. Computer Vision Foundation / IEEE,
work page 2019
-
[9]
doi: 10.1016/J.NEUNET.2019.01.012. M. Raghu, C. Zhang, J. M. Kleinberg, and S. Bengio. Transfusion: Understanding transfer learning for medical imaging. In H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alch´ e-Buc, E. B. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Process...
- [10]
-
[12]
J. Yosinski, J. Clune, Y. Bengio, and H. Lipson. How transferable are features in deep neural networks? In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada ,...
work page 2014
-
[13]
We report other pre-trained networks considered in the paper under different training configuration
13 A Results on other pre-trained networks. We report other pre-trained networks considered in the paper under different training configuration. Table 6: Performance of Gowal et al. (2020) under different configurations. We randomized the last layer and fine-tuned the network by only training this layer. τ = 0.1, η= 0.1 τ = 0.01, η= 0.1 τ = 0.1, η= 1 τ = ...
work page 2020
- [2015]
-
[2016]
doi: 10.1007/978-3-319-46493-0 \
Show all 13 references
-
[2017]
2017.316
doi: 10.1109/CVPR. 2017.316. R. Volpi, H. Namkoong, O. Sener, J. C. Duchi, V. Murino, and S. Savarese. Generalizing to unseen domains via adversarial data augmentation. Advances in Neural Information Processing Systems , 31,
2017 doi
-
[2018]
doi: 10.18653/V1/P18-1031. R. Huang, J. Huang, W. Liu, and H. Ding. Coresets for Wasserstein distributionally robust optimization problems. In Advances in Neural Information Processing Systems , Oct
-
[2019]
Li and D
Z. Li and D. Hoiem. Learning without forgetting. In B. Leibe, J. Matas, N. Sebe, and M. Welling, editors, Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV , volume 9908 of Lecture Notes in Computer Sc...
2016
-
[2021]
J. Cui, Z. Tian, Z. Zhong, X. Qi, B. Yu, and H. Zhang. Decoupled kullback-leibler divergence loss. arXiv preprint arXiv:2305.13948 ,
-
[2022]
Carlini and D
N. Carlini and D. Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy (SP) , pages 39–57. IEEE, May
2017
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.