Pith. sign in

REVIEW 3 major objections 6 minor 52 references

Towards Million-Scale Adversarial Robustness Evaluation With Stronger Individual Attacks

T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Probability-space margin attack beats every individual attack tested.

desk verdict PMA is a real but modest attack improvement; the million-scale robustness claim is confounded by dataset shift and needs controls. read the letter →

arxiv 2411.15210 v4 pith:FAQMAXLW submitted 2024-11-20 cs.LG cs.AIcs.CRcs.CV

Yong Xie , Weijie Zheng , Hanxun Huang , Guangnan Ye , Xingjun Ma This is my paper · ORCID
classification cs.LGcs.AIcs.CRcs.CV
keywords adversarialrobustnesswhite-boxattackprobabilitymarginlossevaluationmillion-scalerelativeCIFAR-10ImageNet
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Probability Margin Attack (PMA) tries to establish that measuring an adversarial margin in probability space, rather than logits space, makes an individual white-box attack reliably stronger. Across 29 leading adversarially trained models on CIFAR-10, CIFAR-100, and ImageNet, PMA reports robust accuracy 0.03% to 0.55% lower than the best existing individual attack, and when paired with one targeted attack it outperforms the standard four-method ensemble at 25% of the runtime. The paper also builds a million-image evaluation set, CC1M, and reports that the relative robustness of top ImageNet models collapses from above 59% on the standard test set to below 19% at million scale. If these results hold, standard benchmark numbers overstate real-world robustness, and cheap individual attacks can replace expensive ensembles in large-scale evaluation.

What carries the argument

The load-bearing object is the probability margin loss $L_{pm}=p_{\max}-p_y$, which replaces the logits difference $z_{\max}-z_y$ of the classic margin loss with a softmax-probability difference. Its gradient decomposes as $p_y\nabla L_{ce} + p_{\max}\nabla L_{ce}^{\mathrm{targeted}}$, making it a single formula that behaves as an adaptive blend of untargeted and targeted cross-entropy. PMA wraps this loss in a two-stage attack that alternates between maximizing $p_{\max}$ and minimizing $p_y$ in stage one, then optimizes the full difference in stage two, with a cosine step-size schedule and multiple restarts; the alternation is what lets the attack explore different wrong-class directions before committing to the full margin.

What would settle it

Take a random 50,000-image subset of CC1M and a 50,000-image subset of the ImageNet test set, run the same PMA and PGD-CE attacks with the same pseudo-label relative-robustness metric on both; if the relative robustness on the CC1M subset is close to the ImageNet subset or the claimed >59%-to-<19% gap disappears, the scale effect has not been isolated from the dataset shift.

Watch

Extended reading notes

Core claim

The central claim is that the probability margin $L_{pm}(z,y)=p_{\max}-p_y$, the gap between the top non-true-class softmax probability and the true-class probability, is a better adversarial objective than logits margin or cross-entropy. The paper's key identity is that its gradient is a probability-weighted combination of the untargeted and targeted cross-entropy gradients, $\nabla_x L_{pm}=p_y\nabla_x L_{ce}+p_{\max}\nabla_x L_{ce}^{t}$, so the attack inherits both broad exploration and focus on the strongest wrong class. Built into a two-stage margin-decomposition pipeline with alternating objectives and restarts, PMA is reported as the strongest individual white-box attack on the 29-model test bed. The paper further claims that the one-attack combination, PMA+APGDT, beats the standard ensemble attack on all tested models while taking about a quarter of the runtime, and that million-scale evaluation on CC1M reveals a far lower robustness than the ImageNet test set suggests.

Load-bearing premise

The million-scale finding assumes that relative robustness measured with model-predicted labels on the filtered CC1M images is a fair proxy for real-world robustness, so the large observed drop is due to evaluation scale rather than to the different image distribution or the filtering procedure.

Editorial extensions

If this is right

  • PMA lowers the measured robust accuracy of leading models by 0.03% to 0.55% compared with the best existing individual attack, so previously reported individual-attack numbers are slightly optimistic.
  • The PMA+APGDT ensemble achieves lower robust accuracy than the standard four-method ensemble on every tested model with about 25% of its runtime, making strong evaluation much cheaper.
  • The CC1M million-scale evaluation puts relative robustness of top ImageNet models below 19%, versus above 59% on the ImageNet test set, implying small-scale leaderboard evaluations substantially overstate robustness on broader data.
  • PMA is roughly 3.56% to 11.86% lower in relative robustness than PGD with cross-entropy on CC1M, so the gap between attacks grows at scale.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: Because the gradient identity expresses the PM loss as a positive combination of two CE gradients, a testable extension is whether any weight pair beyond $(p_y, p_{\max})$ yields even stronger attacks; the paper's own ablation only varies a single scalar weight.
  • Editorial inference: The million-scale comparison conflates scale with distribution shift: a 50k-image control drawn from CC1M and evaluated with the same pseudo-label metric would isolate whether the drop from >59% to <19% is caused by the number of images or by the CC1M distribution itself.
  • Editorial inference: Relative robustness with model-generated pseudo-labels is a weaker metric than ground-truth robustness; if pseudo-labels drift on out-of-distribution images, the reported gap may partly reflect label noise rather than true vulnerability.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes Probability Margin Attack (PMA), a white-box individual attack that uses a probability margin loss within a two-stage margin-decomposition pipeline. It claims that PMA outperforms existing individual attacks on 29 RobustBench models, and that the PMA+APGDT ensemble (PMA+) outperforms AutoAttack while taking only 25% of its runtime. The paper also constructs CC1M, a filtered 1M-image subset of CC3M, and evaluates a newly defined 'relative robustness' metric on ImageNet-robust models, reporting that relative robustness drops from above 59% on ImageNet-1k to below 19% on CC1M, which the authors interpret as a small-scale versus large-scale robustness gap.

Significance. The PMA loss is a simple and well-motivated modification of existing margin-based losses, and the derivation of its gradient as a weighted combination of targeted and untargeted cross-entropy gradients is correct. The empirical pattern across 29 models is consistent: PMA lowers robust accuracy compared with the best baseline in nearly every row, and the efficiency gain over AutoAttack is substantial. If the attack results hold, PMA and PMA+ are practically useful tools for robustness evaluation. The million-scale dataset and the relative robustness metric are steps toward larger-scale evaluation, but the current experimental design does not isolate the effect of scale from distribution shift, and the metric itself is not calibrated without clean-accuracy reporting.

major comments (3)
  1. [§4.4, Figure 1] Section 4.4 compares relative robustness on the ImageNet-1k test set (50k, in-distribution) with CC1M (1M, out-of-distribution web images). Because robust accuracy is a population average, a larger i.i.d. sample from the same distribution would not move the estimate by roughly 40 percentage points; the observed drop from above 59% to below 19% must therefore be driven by the change in image distribution, the pseudo-label protocol, or the LID/MAD filtering. The paper does not provide a same-size (e.g., 50k) subset control drawn from CC1M, does not report clean top-1 accuracy on CC1M, and does not give numeric LID/MAD thresholds. As a result, the claim of a small-scale versus large-scale robustness gap is not established. Please add a subset control and clean-accuracy reporting, or reframe the claim as an out-of-distribution robustness gap.
  2. [§4.4, Relative Robustness definition] The 'relative robustness' metric counts an attack as successful whenever the model's prediction on the adversarial image differs from its own prediction on the clean image, regardless of whether the clean prediction was correct. An attack that flips an incorrect prediction to the correct class is therefore scored as a success, and the metric is strongly coupled to the model's clean accuracy on the test distribution. Without reporting clean top-1 accuracy on CC1M, the stated 3.56% to 11.86% gap between PMA and PGDce on CC1M cannot be interpreted as an adversarial-robustness effect; it may partly reflect lower confidence on out-of-distribution inputs. Please report clean accuracy on CC1M and, if feasible, standard robust accuracy on a labeled subset.
  3. [Tables 3 and 10] There are inconsistencies between the main text and the appendix. For Swin-B (ImageNet), Table 3 reports PMA robust accuracy of 54.41%, while Table 10 reports the SGD+sign PMA value as 54.57%. For RWRN-70-16[27] (CIFAR-10), Table 2 reports PGDpm as 71.76%, while Table 10 reports the SGD+sign PGDpm as 71.16%. These discrepancies affect the exact claims about the magnitude of improvement and must be reconciled or explained (e.g., different seeds, perturbation budgets, or evaluation subsets).
minor comments (6)
  1. [Section 3.1, Eq. (2)] The notation 'zπi represents the i-th largest logit value (except zy)' is ambiguous; please define π explicitly (e.g., π1 is the index of the maximum non-true-class logit) in the text.
  2. [Algorithm 1] The cosine step-size schedule appears only in Algorithm 1; the main text in Section 3.2 should mention that PMA uses a cosine-decayed step size to make the method self-contained.
  3. [Tables 4 and 5] Tables 4 and 5 are visually dense, with numbers and signs run together; reformatting them into clearer sub-columns would substantially improve readability.
  4. [Section 4.2] The statement 'PMA is the strongest individual attack in the current literature' is stronger than what the experiments show, given the finite set of baselines (though AAA and ACG are covered in the appendix); consider phrasing this as 'strongest among the evaluated individual attacks.'
  5. [General] Several typos appear in the abstract and introduction, including 'defeines' and 'aslo'; a copy-edit is recommended.
  6. [Tables 2, 3, and 6] The small robustness differences (often below 0.1 percentage points) are reported without error bars or statistical tests; a paired test or confidence intervals would help support the claim of consistent improvement.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: PMA's loss is defined independently and benchmarked against external baselines; the CC1M gap is confounded by distribution shift but not forced by the paper's definitions.

full rationale

The paper's central attack contribution is the Probability Margin (PM) loss, defined in Eq. (2) directly from softmax probabilities. No parameter of this loss is fitted to robust-accuracy data; the gradient identity in Eq. (3) is a mathematical equivalence between the PM gradient and a weighted sum of untargeted and targeted cross-entropy gradients, not an empirical prediction. The RobustBench comparisons in Tables 2 and 3 are standard external benchmark evaluations, and PMA is also compared against AutoAttack, MD, ACG, and AAA baselines. The only notable self-citation is the Margin Decomposition (MD) attack [7], which shares authors with this paper and whose pipeline PMA adopts; however, the paper states this explicitly ('we build on the attack pipeline of the Margin Decomposition (MD) attack') and MD is itself used as a baseline, so the citation does not carry the central claim. Hyperparameters K'=25 and n=1 were chosen by ablations on a subset of the same model family (Appendix B), which is a mild selection-on-benchmark concern; the reported ablations show differences of only about 0.01-0.05 for K' and 0.1-0.5 for n, so the headline superiority of PMA is not an artifact of this choice. For the million-scale evaluation, Section 4.4 defines Relative Robustness using the model's own clean predictions as labels and explicitly notes that it 'is a weaker robustness measurement than the standard robustness as it also considers the incorrect predictions.' The large drop from 'above 59% to below 19%' on CC1M is therefore an empirical claim comparing two different datasets (ImageNet-1k vs CC1M) and two different label protocols (ground truth vs pseudo-labels); it is confounded by distribution shift and clean-accuracy differences, and the paper lacks a same-size i.i.d. control that would isolate scale. That is a correctness/rigor limitation, not a circular reduction: the metric is disclosed, the drop is not definitionally forced, and no equation or fitted parameter renders the conclusion equivalent to its inputs. Overall, no step in the derivation chain reduces to its own inputs, so the circularity score is low despite the methodological caveats.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The attack itself introduces no invented entities. The main free parameters are attack hyperparameters and dataset filtering thresholds. The most consequential assumptions are the pseudo-label metric and the use of CC1M as a stand-in for large-scale deployment data, both of which are disclosed but not validated against a same-scale ImageNet control.

free parameters (4)
  • K' (first stage steps) = 25
    Tuned on the same RobustBench evaluation models; best among 15, 20, 25, 30, 35 in Appendix Table 7.
  • Number of restarts n = 1
    Tuned on the same evaluation models; best among 1, 2, 5 in Appendix Table 8.
  • CC1M LID/MAD selection threshold = Not specified
    The paper selects 1 million images with LID scores close to the median without giving a numeric MAD cutoff, so the dataset composition is not reproducible.
  • PM loss weight beta = 1
    PMA is best at beta=1, but PGDpm is slightly better at beta=0.75 (Appendix Table 12); the paper keeps beta=1 in all main experiments.
assumptions (4)
  • standard math Softmax identities and the chain rule for gradient computation
    Used in Section 3.1 Eq. (3) to express the PM loss gradient as a weighted combination of CE gradients.
  • domain assumption RobustBench leaderboard models are a representative sample of state-of-the-art defenses
    All attack comparisons use the top RobustBench models; the 'strongest attack' conclusion is bounded by this model set.
  • ad hoc to paper Relative robustness with pseudo-labels is a valid evaluation metric on unlabeled data
    Section 4.4 defines the metric; the paper itself notes it is weaker than standard robustness.
  • domain assumption CC3M-derived CC1M after filtering is a meaningful large-scale distribution for ImageNet models
    The million-scale evaluation assumes that the filtered Conceptual Captions images are a fair proxy for deployment; the domain mismatch is only discussed in the CLIP supplementary experiments.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Million-Scale Adversarial Robustness Evaluation With Stronger Individual Attacks." pith.science (2026). https://pith.science/paper/FAQMAXLW

@misc{pith2026241115210,
  author       = {Pith},
  title        = {Pith review of: Towards Million-Scale Adversarial Robustness Evaluation With Stronger Individual Attacks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FAQMAXLW}},
  note         = {Machine review of arXiv:2411.15210}
}
read the original abstract

As deep learning models are increasingly deployed in safety-critical applications, evaluating their vulnerabilities to adversarial perturbations is essential for ensuring their reliability and trustworthiness. Over the past decade, a large number of white-box adversarial robustness evaluation methods (i.e., attacks) have been proposed, ranging from single-step to multi-step methods and from individual to ensemble methods. Despite these advances, challenges remain in conducting meaningful and comprehensive robustness evaluations, particularly when it comes to large-scale testing and ensuring evaluations reflect real-world adversarial risks. In this work, we focus on image classification models and propose a novel individual attack method, Probability Margin Attack (PMA), which defines the adversarial margin in the probability space rather than the logits space. We analyze the relationship between PMA and existing cross-entropy or logits-margin-based attacks, and show that PMA can outperform the current state-of-the-art individual methods. Building on PMA, we propose two types of ensemble attacks that balance effectiveness and efficiency. Furthermore, we create a million-scale dataset, CC1M, derived from the existing CC3M dataset, and use it to conduct the first million-scale white-box adversarial robustness evaluation of adversarially-trained ImageNet models. Our findings provide valuable insights into the robustness gaps between individual versus ensemble attacks and small-scale versus million-scale evaluations.

Figures

Figures reproduced from arXiv: 2411.15210 by the authors.

Figure 1
Figure 1. Relative robustness evaluation on CC1M and ImageNet [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Visual illustrations of the attacked images. [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

52 extracted references · 40 canonical work pages

  1. [27]

    Robust principles: Architectural design principles for adversarially robust cnns,

    S. Peng, W. Xu, C. Cornelius, M. Hull, K. Li, R. Duggal, M. Phute, J. Martin, and D. H. Chau, “Robust principles: Architectural design principles for adversarially robust cnns,” arXiv preprint arXiv:2308.16258, 2023. 5, 6, 7, 8, 1, 2

    arXiv 2023

  2. [1]

    Intriguing properties of neural networks,

    C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus, “Intriguing properties of neural networks,” arXiv preprint arXiv:1312.6199, 2013. 1, 2

    arXiv 2013

  3. [2]

    Explain- ing and harnessing adversarial examples,

    I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explain- ing and harnessing adversarial examples,” arXiv preprint arXiv:1412.6572, 2014. 1, 2

    arXiv 2014

  4. [3]

    Towards evaluating the robustness of neural networks,

    N. Carlini and D. Wagner, “Towards evaluating the robustness of neural networks,” in IEEES&P, 2017. 1, 2, 3, 4

    work page 2017

  5. [4]

    Towards deep learning models resistant to adversarial attacks,

    A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,” arXiv preprint arXiv:1706.06083, 2017. 1, 2, 4

    arXiv 2017

  6. [5]

    Reliable evaluation of adversarial ro- bustness with an ensemble of diverse parameter-free attacks,

    F. Croce and M. Hein, “Reliable evaluation of adversarial ro- bustness with an ensemble of diverse parameter-free attacks,” in ICML, 2020. 1, 2, 3, 4, 5

    work page 2020

  7. [6]

    Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples,

    A. Athalye, N. Carlini, and D. Wagner, “Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples,” in ICML, 2018. 1, 2

    work page 2018

  8. [7]

    Imbalanced gradients: a subtle cause of overestimated adversarial robustness,

    X. Ma, L. Jiang, H. Huang, Z. Weng, J. Bailey, and Y .-G. Jiang, “Imbalanced gradients: a subtle cause of overestimated adversarial robustness,” Machine Learning, pp. 1–26, 2023. 1, 2, 3, 4

    work page 2023

Show all 52 references
  1. [8]

    Practical evaluation of adversarial robustness via adaptive auto attack,

    Y . Liu, Y . Cheng, L. Gao, X. Liu, Q. Zhang, and J. Song, “Practical evaluation of adversarial robustness via adaptive auto attack,” in CVPR, 2022. 1, 2, 5

    2022

  2. [9]

    Diversity can be trans- ferred: Output diversification for white-and black-box at- tacks,

    Y . Tashiro, Y . Song, and S. Ermon, “Diversity can be trans- ferred: Output diversification for white-and black-box at- tacks,” in NeurIPS, 2020. 1, 2

    2020

  3. [10]

    An alternative surrogate loss for pgd-based adver- sarial testing,

    S. Gowal, J. Uesato, C. Qin, P.-S. Huang, T. Mann, and P. Kohli, “An alternative surrogate loss for pgd-based adver- sarial testing,” arXiv preprint arXiv:1910.09338, 2019. 1, 2

    1910 arXiv

  4. [11]

    Lafeat: Piercing through adversarial defenses with latent features,

    Y . Yu, X. Gao, and C.-Z. Xu, “Lafeat: Piercing through adversarial defenses with latent features,” in CVPR, 2021. 1, 2

    2021

  5. [12]

    Alternating objectives generates stronger pgd-based adversarial attacks,

    N. Antoniou, E. Georgiou, and A. Potamianos, “Alternating objectives generates stronger pgd-based adversarial attacks,” arXiv preprint arXiv:2212.07992, 2022. 1, 2, 4

    2022 arXiv

  6. [13]

    Efficient loss function by minimizing the detrimental effect of floating-point errors on gradient-based attacks,

    Y . Yu and C.-Z. Xu, “Efficient loss function by minimizing the detrimental effect of floating-point errors on gradient-based attacks,” in CVPR, 2023. 1, 2, 4

    2023

  7. [14]

    Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning,

    P. Sharma, N. Ding, S. Goodman, and R. Soricut, “Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning,” in ACL, 2018. 2, 7

    2018

  8. [15]

    Adversarial exam- ples in the physical world,

    A. Kurakin, I. Goodfellow, and S. Bengio, “Adversarial exam- ples in the physical world,” arXiv preprint arXiv:1607.02533,

    arXiv

  9. [16]

    Theoretically principled trade-off between robustness and accuracy,

    H. Zhang, Y . Yu, J. Jiao, E. Xing, L. El Ghaoui, and M. Jordan, “Theoretically principled trade-off between robustness and accuracy,” inICML, 2019. 2

    2019

  10. [17]

    Minimally distorted adversarial ex- amples with a fast adaptive boundary attack,

    F. Croce and M. Hein, “Minimally distorted adversarial ex- amples with a fast adaptive boundary attack,” in ICML, 2020. 2, 4

    2020

  11. [18]

    Square attack: a query-efficient black-box adversarial attack via random search,

    M. Andriushchenko, F. Croce, N. Flammarion, and M. Hein, “Square attack: a query-efficient black-box adversarial attack via random search,” in ECCV, 2020. 2

    2020

  12. [19]

    Diversified adversarial attacks based on conjugate gradient method,

    K. Yamamura, H. Sato, N. Tateiwa, N. Hata, T. Mitsutake, I. Oe, H. Ishikura, and K. Fujisawa, “Diversified adversarial attacks based on conjugate gradient method,” in International Conference on Machine Learning. PMLR, 2022, pp. 24 872– 24 894. 2, 5

    2022

  13. [20]

    Robust- bench: a standardized adversarial robustness benchmark,

    F. Croce, M. Andriushchenko, V . Sehwag, E. Debenedetti, N. Flammarion, M. Chiang, P. Mittal, and M. Hein, “Robust- bench: a standardized adversarial robustness benchmark,” in NeurIPS, 2021. 4

    2021

  14. [21]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in CVPR, 2016, pp. 770–778. 4

    2016

  15. [22]

    Wide residual networks,

    S. Zagoruyko and N. Komodakis, “Wide residual networks,” arXiv preprint arXiv:1605.07146, 2016. 4

    2016 arXiv

  16. [23]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929, 2020. 4

    2010 arXiv

  17. [24]

    Xcit: Cross-covariance image transformers,

    A. Ali, H. Touvron, M. Caron, P. Bojanowski, M. Douze, A. Joulin, I. Laptev, N. Neverova, G. Synnaeve, J. Ver- beek et al., “Xcit: Cross-covariance image transformers,” in NeurIPS, 2021. 4

    2021

  18. [25]

    Swin transformer: Hierarchical vision transformer using shifted windows,

    Z. Liu, Y . Lin, Y . Cao, H. Hu, Y . Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in ICCV, 2021. 4

    2021

  19. [26]

    A convnet for the 2020s,

    Z. Liu, H. Mao, C.-Y . Wu, C. Feichtenhofer, T. Darrell, and S. Xie, “A convnet for the 2020s,” inCVPR, 2022. 4

    2022

  20. [28]

    Better diffusion models further improve adversarial training,

    Z. Wang, T. Pang, C. Du, M. Lin, W. Liu, and S. Yan, “Better diffusion models further improve adversarial training,” in ICML, 2023. 5, 6, 7, 8, 1, 2

    2023

  21. [29]

    Improving the accuracy-robustness trade-off of classifiers via adaptive smoothing,

    Y . Bai, B. G. Anderson, A. Kim, and S. Sojoudi, “Improving the accuracy-robustness trade-off of classifiers via adaptive smoothing,” arXiv preprint arXiv:2301.12554, 2023. 5, 6, 7, 8, 1

    2023 arXiv

  22. [30]

    Decoupled kullback-leibler divergence loss,

    J. Cui, Z. Tian, Z. Zhong, X. Qi, B. Yu, and H. Zhang, “Decoupled kullback-leibler divergence loss,”arXiv preprint arXiv:2305.13948, 2023. 5, 6, 7, 8, 1, 2

    2023 arXiv

  23. [31]

    Fixing data augmentation to improve adversar- ial robustness,

    S.-A. Rebuffi, S. Gowal, D. A. Calian, F. Stimberg, O. Wiles, and T. Mann, “Fixing data augmentation to improve adversar- ial robustness,” arXiv preprint arXiv:2103.01946, 2021. 5, 6, 7, 8, 1

    2021 arXiv

  24. [32]

    Improving robustness using generated data,

    S. Gowal, S.-A. Rebuffi, O. Wiles, F. Stimberg, D. A. Calian, and T. A. Mann, “Improving robustness using generated data,” in NeurIPS, 2021. 5, 6, 7, 8

    2021

  25. [33]

    Uncover- ing the limits of adversarial training against norm-bounded ad- versarial examples,

    S. Gowal, C. Qin, J. Uesato, T. Mann, and P. Kohli, “Uncover- ing the limits of adversarial training against norm-bounded ad- versarial examples,” arXiv preprint arXiv:2010.03593, 2020. 5, 6, 7, 8, 1

    2010 arXiv

  26. [34]

    Revisiting residual networks for adversarial robustness,

    S. Huang, Z. Lu, K. Deb, and V . N. Boddeti, “Revisiting residual networks for adversarial robustness,” in CVPR, 2023. 5, 6, 7, 8

    2023

  27. [35]

    A light recipe to train robust vision transformers,

    E. Debenedetti, V . Sehwag, and P. Mittal, “A light recipe to train robust vision transformers,” in SaTML, 2023. 5, 6, 8 9

    2023

  28. [36]

    Robustness and accuracy could be reconcilable by (proper) definition,

    T. Pang, M. Lin, X. Yang, J. Zhu, and S. Yan, “Robustness and accuracy could be reconcilable by (proper) definition,” in ICML, 2022. 5, 6, 8

    2022

  29. [37]

    A comprehensive study on robustness of image classification models: Benchmarking and rethinking,

    C. Liu, Y . Dong, W. Xiang, X. Yang, H. Su, J. Zhu, Y . Chen, Y . He, H. Xue, and S. Zheng, “A comprehensive study on robustness of image classification models: Benchmarking and rethinking,” arXiv preprint arXiv:2302.14301, 2023. 5, 6, 8, 1, 2, 3

    2023 arXiv

  30. [38]

    Mixednuts: Training-free accuracy-robustness balance via nonlinearly mixed classifiers,

    Y . Bai, M. Zhou, V . M. Patel, and S. Sojoudi, “Mixednuts: Training-free accuracy-robustness balance via nonlinearly mixed classifiers,” arXiv preprint arXiv:2402.02263, 2024. 5, 6, 8, 2

    2024 arXiv

  31. [39]

    Revisiting adversarial training for imagenet: Architectures, training and general- ization across threat models,

    N. D. Singh, F. Croce, and M. Hein, “Revisiting adversarial training for imagenet: Architectures, training and general- ization across threat models,” in NeurIPS, 2024. 5, 6, 8, 1, 2

    2024

  32. [40]

    Local intrinsic dimensionality I: an extreme- value-theoretic foundation for similarity applications,

    M. E. Houle, “Local intrinsic dimensionality I: an extreme- value-theoretic foundation for similarity applications,” in SISAP, 2017. 7

    2017

  33. [41]

    Char- acterizing adversarial subspaces using local intrinsic dimen- sionality,

    X. Ma, B. Li, Y . Wang, S. M. Erfani, S. Wijewickrema, G. Schoenebeck, M. E. Houle, D. Song, and J. Bailey, “Char- acterizing adversarial subspaces using local intrinsic dimen- sionality,” inICLR, 2018. 7

    2018

  34. [42]

    Collider: A robust training framework for backdoor data,

    H. M. Dolatabadi, S. Erfani, and C. Leckie, “Collider: A robust training framework for backdoor data,” inACCV, 2022. 7

    2022

  35. [43]

    Detecting backdoor samples in contrastive language image pretraining,

    H. Huang, S. Erfani, Y . Li, X. Ma, and J. Bailey, “Detecting backdoor samples in contrastive language image pretraining,” in ICLR, 2025. 7

    2025

  36. [44]

    Ldreg: Local dimensionality regular- ized self-supervised learning,

    H. Huang, R. J. G. B. Campello, S. M. Erfani, X. Ma, M. E. Houle, and J. Bailey, “Ldreg: Local dimensionality regular- ized self-supervised learning,” in ICLR, 2024. 7 10 Towards Million-Scale Adversarial Robustness Evaluation With Stronger Individual Attacks Supplementary Material

    2024

  37. [45]

    Introduction Due to the page limitation of the paper, we further illustrate our method in this supplementary material, which includes the following sections: 1)Visual illustrations of the attacked images. 2) A detailed analysis of the quantitative results for hyperparameters K...

  38. [46]

    Figure 2

    isual illustrations of the attacked images. Figure 2. Visual illustrations of the attacked images. Fig. 2 visualizes adversarial examples generated by dif- ferent attacks, along with their predicted labels. We eval- uate their quality using PSNR, SSIM, and LPIPS, and re- port ...

  39. [47]

    The models’ robustness (%) evaluated on different K ′ values

    Detailed quantitative results of hyperparam- eter K ′ and n Table 7. The models’ robustness (%) evaluated on different K ′ values. The best results are boldfaced. Dataset Model K′= 15K′= 20K′= 25K′= 30K′= 35CIFAR10WRN-28-10[30]67.79 67.76 67.72 67.77 67.77CIFAR10WRN-28-10[28]6...

  40. [48]

    Seven defense models from the CIFAR10 dataset were subjected to a con- straint of 100 attack steps

    Comparison of experimental results be- tween PMA method and AAA, ACG meth- ods In this comparative analysis, we evaluated the AAA and ACG methods alongside our PMA method. Seven defense models from the CIFAR10 dataset were subjected to a con- straint of 100 attack steps. The o...

  41. [49]

    To extend our analysis, this section introduces comparative experiments with optimizer-based approaches, focusing on the widely recognized Adam optimizer

    Comparison of experimental results be- tween traditional SGD+sign update strat- egy and optimizer-based strategies In our preliminary experiments, we adopted the SGD+sign update strategy, forgoing the integration of an optimizer. To extend our analysis, this section introduces...

  42. [50]

    The robustness (%) of the models, evaluated using the PGDpm attack with varying β values, on the CIFAR10 and Ima- geNet datasets

    Detailed ablation results of Pmax and Py weights Table 12. The robustness (%) of the models, evaluated using the PGDpm attack with varying β values, on the CIFAR10 and Ima- geNet datasets. Dataset Model β= 0.5 β= 0.75 β= 1β= 1.25 β= 1.5CIFAR10WRN-28-10[30]68.66 68.4768.47 68.4...

  43. [51]

    We assessed the same set of five ImageNet defense models discussed in the main body of the paper

    Million-Scale adversarial robustness evalu- ation between the PMA method and other methods In this supplementary section, we broaden our comparative analysis by incorporating the PMA and P GDce methods with other existing techniques. We assessed the same set of five ImageNet d...

  44. [52]

    We tested perturbation ranges of 1, 2, and 3, with a batch size of 32

    Supplementary experiments on CLIP To address the domain mismatch between Conceptual- Captions and ImageNet, we conducted supplementary ex- periments using CLIP on CC1M. We tested perturbation ranges of 1, 2, and 3, with a batch size of 32. All sam- ples in a batch—except the t...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.