Pith. sign in

REVIEW 4 major objections 5 minor 38 references

Robustness should be evaluated as a full curve, not a single epsilon; budgeted minimum-norm attack ensembles approximate the worst-case frontier to 98–99.9%, and a curve-area index ranks defenses epsilon-free.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 11:31 UTC pith:LBWC6DLC

load-bearing objection Solid, pool-relative benchmark paper — worth refereeing, but the near-100% AOI scores are less meaningful than they look. the 4 major comments →

arxiv 2607.19855 v1 pith:LBWC6DLC submitted 2026-07-22 cs.LG cs.CR

Adversarial Frontiers: Minimum-Norm Attack Ensembles for Robustness Evaluation

Luca Scionis , Luca Melis , Maura Pintor , Fabio Brau , Ambra Demontis , Giorgio Fumera , Fabio Roli , Battista Biggio This is my paper
classification cs.LG cs.CR
keywords adversarial robustness evaluationrobustness–perturbation curvesminimum-norm attacksattack ensemblesquery budget allocationattack optimality indexdefense optimality indexepsilon-free ranking
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that fixed-budget, single-point robustness evaluation is incomplete: curves for different models can cross, so rankings at one epsilon can flip at another. It replaces the question "how robust is this model at ε" with "how close to the per-sample worst case can an attack pool get under a query budget," formalizing the attack frontier and a greedy budget-allocation algorithm whose ensembles reach 98–99.9% of that frontier on held-out models across ℓ0, ℓ1, ℓ2, and ℓ∞. On most held-out defenses, the ensembles match or exceed the standard fixed-budget attack ensemble at a controllable and often smaller query cost. The companion Defense Optimality Index aggregates each model's gap to the best curve in a defense pool, giving an epsilon-free ranking that is stable across budget tiers. A sympathetic reader would care because it offers practitioners a principled trade-off between evaluation cost and tightness, and a way to compare defenses without choosing an operating point.

Core claim

The central claim is that the empirical attack frontier — the per-sample minimum perturbation found by any attack in a pool — can be approximated almost exactly by a small, adaptively budgeted subset of that pool, and that the residual gap is a measurable, budget-controllable quantity. Experiments show the greedy ensembles reach an attack-optimality index of 98.1–99.8% on ℓ0 and 98.3–99.9% on ℓ1–ℓ∞ held-out defenses, improving monotonically with query budget; the ℓ2 ensemble matches or exceeds the fixed-budget standard on all 13 held-out models already at 4k queries, and the ℓ1 ensemble does so on all 13 at 12k. The paper further claims that robustness–perturbation curves can cross at budget

What carries the argument

The carrying object is the empirical attack frontier: the per-sample minimum of distances returned by a pool of minimum-norm attacks, whose induced curve is the tightest robustness estimate the pool can give. On top of it sit two area-based indices — the Attack Optimality Index (normalized area gap of one attack's curve to the frontier) and the Defense Optimality Index (ratio of a model's curve area to the maximum over models) — and the greedy budget-allocation algorithm that assigns query increments to attacks by AOI gain per added query, producing a nested, monotonically tightening family of ensembles. The monotonicity relies on the fact that each attack's distance is an upper bound on the

Load-bearing premise

The load-bearing premise is that the hand-picked pool of white-box, gradient-based minimum-norm attacks finds the true per-sample minimal perturbation; if the strongest attack family for a given defense is missing — black-box, gradient-obfuscation, patch-based, or semantic — then the frontier, the 98–99.9% optimality numbers, the comparison with the fixed-budget ensemble, and the DOI rankings are all optimistic and could invert. The paper's own Section 7 acknowledges this lim

What would settle it

Take any defended model and any sample from the evaluation subset, run the full 12k-query pool, and record the per-sample minimum distance; then run a strong attack of a class absent from the pool, for example a query-efficient black-box attack or one designed for gradient obfuscation. If that outside-pool attack finds a misclassifying perturbation strictly smaller than the pool's minimum on even one sample, then the empirical frontier is not tight and every AOI/DOI value derived from it is an overestimate.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Rankings at a single ε are unreliable: crossing curves mean the leaderboard order can flip at budgets beyond the reference; the paper observes 17 CIFAR-10 defenses moving at least 5 ranks on ℓ2 across the evaluated grid.
  • Evaluation quality becomes a user-chosen budget rather than a fixed pipeline: increasing the query budget from 4k to 12k monotonically raises ensemble AOI, with the largest gain in ℓ1 (+1.12) and the smallest in ℓ∞ (+0.28).
  • The greedy allocations generalize: held-out AOI stays within about 0.4 percentage points of the training models, so ensembles tuned on one model set can evaluate unseen defenses.
  • The ℓ0 norm, previously without a benchmark baseline, can now be evaluated: held-out AOI reaches 98.1–99.8% at 12k queries.
  • A candidate defense can be rejected after the cheapest tier if its 4k curve already lies below the incumbent's 12k curve, because additional queries can only reveal it to be weaker.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Inference — Because every index is pool-relative, extending the pool with black-box, patch-based, or semantic attacks would likely change both the AOI numbers and the DOI rankings; the paper's headline comparisons with the fixed-budget ensemble are against one particular pool, and a broader pool could erode some of the reported margins.
  • Inference — The same min-composition machinery could sandwich the true robustness curve between the empirical frontier (an upper bound on minimal distance) and a certified lower bound, turning the "gap to worst case" from a heuristic into an interval with guarantees; the paper does not propose this.
  • Inference — The epsilon-free perspective might push adversarial training to optimize area under the curve rather than accuracy at a single adversarial budget, changing which defenses are considered state of the art; this is a consequence of switching from point evaluation to curve evaluation, not something the paper tests.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper argues that fixed-ε, predefined-ensemble evaluation (RobustBench/AutoAttack) is fundamentally limited: it samples a single point of the robustness–perturbation curve, gives no evidence of closeness to worst-case attacks, and couples evaluation quality to a non-adjustable cost. The authors propose a curve-based framework built on a pool of minimum-norm attacks (PDPGD, PDGD, FMN, ALMA, DDN, σ-zero, plus their new APGD_min). The 'attack frontier' is the per-sample minimum distance over the pool at full budget; ensemble evaluation is formalized as a budget-constrained frontier-approximation problem (Eq. 8), solved by a greedy query allocation (Algorithm 1) that produces nested ensembles E_4k ⊆ E_8k ⊆ E_12k maximizing the pool-relative Attack Optimality Index. The allocation is selected on 17 training defenses and evaluated on 13 held-out defenses on CIFAR-10/ImageNet across ℓ0–ℓ∞. They report that the ensembles match or exceed AutoAttack's AOI on most held-out models and, at the reference ε, on most ℓ2 (13/13) and ℓ1 (10/13 at 12k) benchmarks, but not ℓ∞ (5/13). A Defense Optimality Index (DOI, Eq. 12) ranks defenses by the ratio of their AUREC to the defense-frontier area and is claimed to be ε-independent and stable across budget tiers.

Significance. Strengths include a coherent formalization, a genuine held-out model split with a narrow generalization gap (Figure 2), nested budget tiers with a monotonic tightening property, a sound early-stopping rule (§5.3), and released code. The demonstration that fixed-ε rankings are unstable (Figure 5) usefully instantiates a known but under-documented concern. If the empirical claims survive a fair AutoAttack baseline, this would be a valuable budget-aware, curve-based evaluation tool. The stress-test concern largely lands, however: the 98–99.9% AOI values and 'match/exceed AutoAttack' statements are relative to an empirical pool that is restricted to white-box, gradient-based attacks (Table 2; §7). Because the AOI normalization (Eq. 5) compresses toward 1 when the pool is weak or saturated, the high values are consistent with a self-referential reference rather than proximity to the true worst case. The held-out split mitigates the overfitting half of the AOI-as-objective-and-metric circularity, but not the pool-relativity. These issues are fixable within the manuscript's scope; they require re-scoping the claims, a fairer AA baseline, and uncertainty quantification.

major comments (4)
  1. [§5.1, Tables 4 and 6] The AA baseline in the AOI comparison is reconstructed by evaluating AA at ten ε values (nine distance percentiles from the authors' own frontier plus ε_ref) and linearly interpolating. Since robustness curves are step functions, this is an uncontrolled approximation whose error can match the claimed effects: ℓ∞ held-out AOI differences vs AA are ~0.1–2 pp (e.g., C3: 99.62 vs 99.75). Because the grid comes from the ensemble's own distances, the comparison is not neutral. The ℓ∞ discrepancy between the two tables sharpens this: at 12k the ensemble matches/exceeds AA's AOI on 10/13 held-out models, but at the reference ε only on 5/13, with the mean difference in AA's favor (−0.2±0.3 pp). The abstract's 'match or exceed AutoAttack on most defenses at every budget tier' is unsupported for ℓ∞. Please use a minimum-norm AA baseline (binary search for per-sample distances) or bound the reconstr
  2. [§2.2 (Eq. 5), §7] AOI is normalized by the gap between the trivial rectangle and the pool's own frontier. A weak or saturated pool shrinks that denominator and compresses every AOI toward 1; Table 4's 96–99.9% values, including AA's, are consistent with this compression. The pool (Table 2) contains only white-box gradient attacks, including the authors' own APGD_min, and §7 concedes gradient-obfuscating or non-differentiable defenses are 'not faithfully evaluated.' The DOI (Eq. 12) is normalized by a defense frontier built from the same pool, so a missing attack family inflates the DOI of exactly the defenses that resist the pool and can reorder Table 7. The held-out split answers the overfitting half of the AOI self-referentiality, but not this pool-relativity. Please qualify the claims as pool-relative throughout (including 'comprehensive pool'/'worst-case' in the abstract) and add a sensitivity check w
  3. [§5.1–5.2, Table 6] The cost comparison is asymmetric. Table 6's Q_AA is the query cost of a single AA evaluation at the reference ε (up to 7,566 queries on ℓ2), but the AOI comparison in Table 4 requires AA to be run at ten ε values (the nine percentiles plus ε_ref) to reconstruct a curve; the total cost of that curve is never reported. The claim of 'fixed and controllable query cost' versus AA's 'variable' cost is therefore not apples-to-apples. Please report the total query cost incurred by each side to produce the reported curve-based estimates, and state explicitly what Q_AA covers.
  4. [§5.1, Tables 5 and 7] Results are on a fixed 1,000-sample subset per model–norm pair, yet Table 7 reports DOI rank differences of a few tenths of a percentage point (e.g., ℓ0 C15 vs C8 at 4k, 60.3 vs 59.8) and asserts rank stability with no uncertainty quantification. With N=1,000, the standard error of a robust-accuracy estimate at p≈0.5 is ~1.5 pp, and this noise propagates into AUREC, AOI, and DOI. Without bootstrap confidence intervals over samples, it is unclear whether the reported 0.1–0.5 pp separations in DOI and the per-model AOI comparisons are signal or noise. Please add confidence intervals or a paired per-sample test for the ensemble-vs-AA comparisons and justify the sample size.
minor comments (5)
  1. [Table 3 and captions] The indented Restarts/Targeted rows report cumulative queries and en-dashes mean inactive groups, but the caption does not explain how group totals relate to the base run; add a worked example. The URL in the caption, 'https://github.com/adversarial-frontier.github.io', appears malformed — it should point to an actual repository (and to the anonymized copy for review).
  2. [§5.1] 'All baseline solvers are executed with their default hyperparameters, The only exception is...' — comma splice; also the 'closed-form ℓ2/3 concave surrogate proximal operator' used for PDPGD in ℓ0 is introduced without a citation. Please cite the surrogate or give its definition.
  3. [References] The reference list contains formatting artifacts (e.g., [1] 'V ol.', [4] missing venue formatting, [5] broken line breaks around the DOI). A cleanup pass is needed.
  4. [§2.2, §3] The term 'attack frontier' is used both for the per-sample minimizer a* and for the frontier curve ρ_a*(ε); the two senses appear together in Eqs. (3)–(5) and in Figure 2's caption. Please use distinct terms or define both explicitly at first use.
  5. [§7] The sentence claiming that black-box, semantic, or patch-based perturbations can be added 'at the sole cost of generalizing the query unit' is unsupported: semantic perturbation families have no natural query-based cost accounting. This should be framed explicitly as future work rather than as a property of the framework.

Circularity Check

0 steps flagged

No significant circularity; the ensemble optimization is self-contained, validated on held-out models, and anchored by an external AutoAttack comparison.

full rationale

The paper's central construction is a greedy query-budget allocation that maximizes the Attack Optimality Index (AOI), which is by definition relative to the empirical attack frontier of a fixed attack pool. This is explicitly disclosed: 'AttackBench [8] provides a quantitative measure of optimality that is, by definition, relative to a predefined pool A of N attacks.' The reported AOI values on held-out models are out-of-sample: the greedy allocation is fit on 17 training models and then evaluated on 13 held-out defenses, so the headline 98-99.9% AOI values are not the fitted objective itself. The comparison against AutoAttack provides an external, non-pool-relative benchmark, and Table 6 reports fixed-epsilon robust accuracy, which is an independent metric not normalized by the pool's frontier. The Defense Optimality Index (Eq. 12) is a defined normalized area, so its epsilon-independence follows by construction, but this is a definitional property rather than a smuggled prediction. Section 7 candidly states that the frontiers are empirical and pool-relative and that the pool contains only white-box gradient-based attacks; this is a validity limitation about the reference frontier, not a circular step. Self-citations to AttackBench and FMN are provenance for definitions and algorithmic techniques that are restated in the paper, and they are not load-bearing for the main empirical claims. Therefore, no specific reduction of a prediction to its own inputs is exhibited.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 1 invented entities

The central new artifacts are the optimized query allocation (a fitted vector per norm/tier) and APGD_min. The main unstated premise is that the hand-picked attack pool is a faithful proxy for the true worst case; the paper discloses this as a limitation. The AOI/DOI metrics are pool-relative by construction.

free parameters (4)
  • Greedy query allocation vector q per norm and budget tier = Table 3 (e.g., ℓ2@12k: Alma=1000, PDGD=1000, APGDmin-DLR=700, APGDmin-CE=600, plus restart/targeted sums)
    Optimized on the 17 training defenses to maximize AOI against the pool frontier. This is the central artifact of the method and is in-sample by construction; held-out evaluation is the only independent check.
  • APGD_min internal hyperparameters = not reported in paper (implementation-only)
    Radius shrink/expand schedule, cosine decay, and checkpointing are inherited from APGD/FMN but no numerical values are given; they could have been tuned on the training models without disclosure, and are unverifiable without code.
  • AutoAttack curve reconstruction grid = 9 frontier distance percentiles + ε_ref, linearly interpolated
    Chosen by hand to build the AA baseline curve for the AOI comparison; interpolation error directly affects the head-to-head AOI numbers.
  • Validation subset size and train/held-out split = 1,000 samples per model-norm; seed 0; 17/13 split
    All rankings and AOI values are computed on this small subset; the split determines which models are in-sample. No confidence intervals are reported.
axioms (6)
  • domain assumption The empirical attack frontier over the chosen white-box gradient attack pool reliably approximates the true worst-case robustness.
    Load-bearing for all 'optimality' claims: AOI and DOI are measured against the pool frontier (Eqs. 5, 11–12). If the pool omits the strongest attack family, AOI/DOI overstate optimality. Acknowledged in Sec. 7 as a limitation.
  • domain assumption Per-sample perturbation distances returned by pool attacks are non-increasing as the query budget grows (best-so-far tracking).
    Used for the monotone-tightening claim and for the early-rejection procedure in Sec. 5.3; true for APGD-min/FMN-style attacks but not proven for every pool member in the paper.
  • domain assumption A 1,000-sample subset per model–norm pair is representative of the full test distribution.
    All tables and rank-shift plots use this subset (Sec. 5.1); no confidence intervals are given, so rank changes of 2–5 positions (Fig. 5) may be sampling noise.
  • domain assumption Linear interpolation between 9 frontier-derived percentiles plus ε_ref reconstructs AutoAttack's robustness curve accurately enough to compute its AOI.
    Introduced in Sec. 5.1. This reconstructed curve is the external baseline in the AOI comparison; any approximation error biases the comparison in favor of the densely sampled ensemble curves.
  • domain assumption Query count (sum of forward and backward passes) is a fair, comparable cost unit across heterogeneous attacks.
    Stated in Sec. 3.1 to make budgets comparable; AutoAttack's curve reconstruction cost is not reported in the same units, complicating the cost comparison.
  • domain assumption The greedy allocation heuristic adequately approximates the integer budgeted maximization in Eq. (8).
    Algorithm 1 has no optimality guarantee (Sec. 3.2); only empirical validation on held-out models supports it.
invented entities (1)
  • APGD_min (minimum-norm Auto-Projected Gradient Descent) independent evidence
    purpose: New minimum-norm adaptation of APGD for ℓ1, ℓ2, and ℓ∞ that searches small perturbations in one run, strengthening the attack pool.
    Described in Sec. 3.3; validated only indirectly through ensemble AOI on held-out models. An independent handle exists via the released code and through direct comparison with FMN, PDPGD, and Alma on the same defenses, but is not isolated in an ablation.

pith-pipeline@v1.3.0-alltime-deepseek · 22947 in / 22855 out tokens · 218111 ms · 2026-08-01T11:31:15.623301+00:00 · methodology

0 comments
read the original abstract

Adversarial robustness is commonly evaluated with predefined attack ensembles, such as AutoAttack, at a single perturbation budget $\varepsilon$ and on a selective choice of perturbation norms. We argue this formulation is fundamentally limited. First, robustness--perturbation curves may intersect or decay at different rates across models, making single-$\varepsilon$ rankings unstable. Second, current ensembles provide no evidence of optimality, leaving an unknown gap to worst-case performance. Third, fixed attack configurations provide no systematic control over the trade-off between attack strength and evaluation cost. To address these limitations, we introduce a unified evaluation framework based on a comprehensive pool of minimum-norm attacks and robustness--perturbation curves across $\ell_0$, $\ell_1$, $\ell_2$ and $\ell_\infty$ norms. We define the attack frontier as the worst-case robustness estimate the attack pool produces against a model. We then formalize evaluation as a frontier-approximation problem, constructing minimum-norm attack ensembles, optimized subsets of the comprehensive pool, that approach the frontier under a controllable query budget, with larger budgets monotonically tightening the estimate. Furthermore, we define the defense frontier as the maximum robustness across the model set at each perturbation size. We finally propose the Defense Optimality Index to rank defenses by their gap to the defense frontier, providing a ranking without selecting a reference $\varepsilon$. On CIFAR-10 and ImageNet, our ensembles match or exceed AutoAttack on most defenses at every budget tier, at fixed and controllable query cost, offering practitioners a query-controlled, curve-based alternative to fixed-$\varepsilon$ evaluation.

Figures

Figures reproduced from arXiv: 2607.19855 by Ambra Demontis, Battista Biggio, Fabio Brau, Fabio Roli, Giorgio Fumera, Luca Melis, Luca Scionis, Maura Pintor.

Figure 1
Figure 1. Figure 1: The limitation of fixed-ε evaluations. This plot shows the robustness–perturbation curves, ρ(ε), for two hypothetical models, whose ro￾bustness decays at different rates. At a fixed reference budget εref, the red model is more robust than the blue one. Yet, the latter appear stronger at larger ε. In Section 5 we demonstrate empirically how single-point evaluations are incom￾plete, motivating the need for ε… view at source ↗
Figure 2
Figure 2. Figure 2: Generalization of attack optimality across query budgets. This plot shows AOI as a function of the query budget Q across the four evaluated perturbation norms. Blue lines denote the Attack Optimality Index on average across the set of 17 training models, while red lines denote the same measure on average across 13 held-out models. The AOI monotonically improves as the budget increases from 4k to 12k querie… view at source ↗
Figure 4
Figure 4. Figure 4: Curves can cross past the reference ε. Robust accuracy ρ(ε) vs. perturbation budget for defense pairs whose curves intersect at a budget larger than AA’s reference ε (dashed line): at the reference one defense leads, beyond it the order flips. Left: ℓ2 on CIFAR-10 (C9/C19); right: the single ℓ∞ case we observe, on ImageNet (I1/I3). We also examine how much increasing the query budget im￾proves AOI on the h… view at source ↗
Figure 5
Figure 5. Figure 5: Rank instability across perturbation budgets. Defense rankings (y-axis) computed via AA across ten different perturbation sizes ε (x-axis). In each panel, colored traces highlight the five models exhibiting the highest rank instability, while the remaining models are shown in gray. The vertical dashed line indicates the standard benchmark ε. the top two under ℓ1 and ℓ2, whereas C10 ranks 19th under ℓ0 but … view at source ↗
Figure 6
Figure 6. Figure 6: Robustness–perturbation curves for all 30 evaluated defenses (E12k). For each dataset, the top row shows the ℓ0 and ℓ1 norms, while the bottom row shows ℓ2 and ℓ∞. The curves plot the robust accuracy against the perturbation size, colored by their DOI score. The red dashed line represents the defense frontier. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_6.png] view at source ↗
Figure 6
Figure 6. Figure 6: The rankings are stable across the evaluated tiers: six of the eight (norm, dataset) combinations retain the same strict order￾ing at 4k, 8k, and 12k. On CIFAR-10, ℓ1 differs only by a C19/C20 tie at 4k, while ℓ0 contains small local swaps at 4k that resolve by 8k. This is empirical stability rather than a guar￾antee, since the frontier and DOI can change with the query budget, as seen under ℓ0. 5.3. Using… view at source ↗
Figure 7
Figure 7. Figure 7: shows one such case on ℓ∞ taken from our model pool: the curve of C9 evaluated at E4k already sits entirely be￾low that of C6 evaluated at E12k, so C9 is less robust than C6 across every perturbation size, and the remaining 8k queries would only widen the gap. The practitioner discards C9 having spent a third of the full budget. If instead the two curves are close or overlap at 4k, the comparison is not ye… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

38 extracted references · 6 linked inside Pith

  1. [1]

    Biggio, I

    B. Biggio, I. Corona, D. Maiorca, B. Nelson, N. Šrndi ´c, P. Laskov, G. Giacinto, F. Roli, Evasion attacks against machine learning at test time, in: Machine Learning and Knowledge Discovery in Databases, V ol. 8190 of Lecture Notes in Computer Science, Springer, 2013, pp. 387–402

    2013

  2. [2]

    Szegedy, W

    C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. J. Goodfellow, R. Fergus, Intriguing properties of neural networks, in: International Conference on Learning Rep- resentations, 2014. 18

    2014

  3. [3]

    Croce, M

    F. Croce, M. Andriushchenko, V . Sehwag, E. Debenedetti, N. Flammarion, M. Chiang, P. Mittal, M. Hein, Robust- Bench: A standardized adversarial robustness benchmark, in: NeurIPS Datasets and Benchmarks, 2021

    2021

  4. [4]

    Croce, M

    F. Croce, M. Hein, Reliable evaluation of adversarial ro- bustness with an ensemble of diverse parameter-free at- tacks, in: International Conference on Machine Learning, 2020

    2020

  5. [5]

    Biggio, F

    B. Biggio, F. Roli, Wild patterns: Ten years after the rise of adversarial machine learn- ing, Pattern Recognition 84 (2018) 317–331. doi:https://doi.org/10.1016/j.patcog.2018.07.023

    doi:10.1016/j.patcog.2018.07.023 2018

  6. [6]

    Carlini, A

    N. Carlini, A. Athalye, N. Papernot, W. Brendel, J. Rauber, D. Tsipras, I. Goodfellow, A. Madry, A. Kurakin, On evaluating adversarial robustness, arXiv preprint arXiv:1902.06705 (2019)

    Pith/arXiv arXiv 1902

  7. [7]

    Risse, C

    N. Risse, C. Göpfert, J. P. Göpfert, How to compare ad- versarial robustness of classifiers from a global perspec- tive, in: International Conference on Artificial Neural Net- works, Springer, 2021, pp. 29–41

    2021

  8. [8]

    A. E. Cinà, J. Rony, M. Pintor, L. Demetrio, A. Demontis, B. Biggio, I. Ben Ayed, F. Roli, AttackBench: Evaluating gradient-based attacks for adversarial examples, Proceed- ings of the AAAI Conference on Artificial Intelligence 39 (3) (2025) 2600–2608

    2025

  9. [9]

    Göpfert, J

    C. Göpfert, J. P. Göpfert, B. Hammer, Adversarial ro- bustness curves, in: Joint European Conference on Ma- chine Learning and Knowledge Discovery in Databases, Springer, 2019, pp. 172–179

    2019

  10. [10]

    Croce, M

    F. Croce, M. Hein, Mind the box:ℓ 1-APGD for sparse adversarial attacks on image classifiers, in: International Conference on Machine Learning, 2021

    2021

  11. [11]

    Pintor, F

    M. Pintor, F. Roli, W. Brendel, B. Biggio, Fast minimum- norm adversarial attacks through adaptive norm con- straints, Advances in Neural Information Processing Sys- tems 34 (2021) 20052–20062

    2021

  12. [12]

    Addepalli, S

    S. Addepalli, S. Jain, G. Sriramanan, R. Venkatesh Babu, Scaling adversarial training to large perturbation bounds, in: European Conference on Computer Vision, Springer, 2022, pp. 301–316

    2022

  13. [13]

    Rade, S.-M

    R. Rade, S.-M. Moosavi-Dezfooli, Helper-based adversar- ial training: Reducing excessive margin to achieve a better accuracy vs. robustness trade-off, in: ICML 2021 Work- shop on Adversarial Machine Learning, 2021

    2021

  14. [14]

    Chen, C.-R

    E.-C. Chen, C.-R. Lee, Data filtering for efficient adver- sarial training, Pattern Recognition 151 (2024) 110394

    2024

  15. [15]

    Augustin, A

    M. Augustin, A. Meinke, M. Hein, Adversarial robustness on in-and out-distribution improves explainability, in: Eu- ropean Conference on Computer Vision, Springer, 2020, pp. 228–245

    2020

  16. [16]

    Rebuffi, S

    S.-A. Rebuffi, S. Gowal, D. A. Calian, F. Stimberg, O. Wiles, T. Mann, Fixing data augmentation to improve adversarial robustness, arXiv preprint arXiv:2103.01946 (2021)

    Pith/arXiv arXiv 2021

  17. [17]

    Debenedetti, V

    E. Debenedetti, V . Sehwag, P. Mittal, A light recipe to train robust vision transformers, in: 2023 IEEE conference on secure and trustworthy machine learning (SaTML), IEEE, 2023, pp. 225–253

    2023

  18. [18]

    Sehwag, S

    V . Sehwag, S. Mahloujifar, T. Handina, S. Dai, C. Xiang, M. Chiang, P. Mittal, Robust learning meets generative models: Can proxy distributions improve adversarial ro- bustness?, arXiv preprint arXiv:2104.09425 (2021). 19

    Pith/arXiv arXiv 2021

  19. [19]

    Rodríguez-Muñoz, T

    A. Rodríguez-Muñoz, T. Wang, A. Torralba, Characteriz- ing model robustness via natural input gradients, in: Eu- ropean Conference on Computer Vision, Springer, 2024, pp. 161–178

    2024

  20. [20]

    J. Cui, Z. Tian, Z. Zhong, X. Qi, B. Yu, H. Zhang, Decou- pled kullback-leibler divergence loss, Advances in Neural Information Processing Systems 37 (2024) 74461–74486

    2024

  21. [21]

    Stutz, M

    D. Stutz, M. Hein, B. Schiele, Confidence-calibrated ad- versarial training: Generalizing to unseen attacks, in: In- ternational conference on machine learning, PMLR, 2020, pp. 9155–9166

    2020

  22. [22]

    N. D. Singh, F. Croce, M. Hein, Revisiting adversarial training for imagenet: Architectures, training and gener- alization across threat models, Advances in Neural Infor- mation Processing Systems 36 (2023) 13931–13955

    2023

  23. [23]

    Zhong, Y

    X. Zhong, Y . Huang, C. Liu, Towards efficient training and evaluation of robust models againstℓ 0 bounded ad- versarial perturbations, in: Forty-first International Con- ference on Machine Learning, 2024

    2024

  24. [24]

    E. Wong, L. Rice, J. Z. Kolter, Fast is better than free: Revisiting adversarial training, arXiv preprint arXiv:2001.03994 (2020)

    Pith/arXiv arXiv 2001

  25. [25]

    Gowal, S.-A

    S. Gowal, S.-A. Rebuffi, O. Wiles, F. Stimberg, D. A. Calian, T. A. Mann, Improving robustness using generated data, Advances in neural information processing systems 34 (2021) 4218–4233

    2021

  26. [26]

    T. Pang, M. Lin, X. Yang, J. Zhu, S. Yan, Robustness and accuracy could be reconcilable by (proper) definition, in: International conference on machine learning, PMLR, 2022, pp. 17258–17277

    2022

  27. [27]

    Salman, A

    H. Salman, A. Ilyas, L. Engstrom, A. Kapoor, A. Madry, Do adversarially robust imagenet models transfer better?, Advances in Neural Information Processing Systems 33 (2020) 3533–3545

    2020

  28. [28]

    Jiang, C

    Y . Jiang, C. Liu, Z. Huang, M. Salzmann, S. Susstrunk, Towards stable and efficient adversarial training againstℓ1 bounded adversarial attacks, in: International Conference on Machine Learning, PMLR, 2023, pp. 15089–15104

    2023

  29. [29]

    Maini, E

    P. Maini, E. Wong, Z. Kolter, Adversarial robustness against the union of multiple perturbation models, in: International Conference on Machine Learning, PMLR, 2020, pp. 6640–6650

    2020

  30. [30]

    Z. Wang, T. Pang, C. Du, M. Lin, W. Liu, S. Yan, Bet- ter diffusion models further improve adversarial training, in: International conference on machine learning, PMLR, 2023, pp. 36246–36263

    2023

  31. [31]

    Y . Xu, Y . Sun, M. Goldblum, T. Goldstein, F. Huang, Ex- ploring and exploiting decision boundary dynamics for adversarial robustness, arXiv preprint arXiv:2302.03015 (2023)

    Pith/arXiv arXiv 2023

  32. [32]

    Matyasko, L.-P

    A. Matyasko, L.-P. Chau, PDPGD: Primal-dual proxi- mal gradient descent adversarial attack, arXiv preprint arXiv:2106.01538 (2021)

    Pith/arXiv arXiv 2021

  33. [33]

    J. Rony, E. Granger, M. Pedersoli, I. Ben Ayed, Aug- mented lagrangian adversarial attacks, in: Proceedings of the IEEE/CVF International Conference on Computer Vi- sion, 2021, pp. 7738–7747

    2021

  34. [34]

    J. Rony, L. G. Hafemann, L. S. Oliveira, I. Ben Ayed, R. Sabourin, E. Granger, Decoupling direction and norm for efficient gradient-basedℓ 2 adversarial attacks and de- fenses, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 4322–4330. 20

    2019

  35. [35]

    A. E. Cinà, F. Villani, M. Pintor, L. Schönherr, B. Big- gio, M. Pelillo,σ-zero: Gradient-based optimization of ℓ0-norm adversarial examples, in: The Thirteenth Interna- tional Conference on Learning Representations, 2025

    2025

  36. [36]

    C. Yao, P. Bielik, P. Tsankov, M. Vechev, Automated discovery of adaptive attacks on adversarial defenses, Advances in Neural Information Processing Systems 34 (2021) 26858–26870

    2021

  37. [37]

    S. Liu, F. Peng, K. Tang, Reliable robustness evaluation via automatically constructed attack ensembles, in: Pro- ceedings of the AAAI Conference on Artificial Intelli- gence (AAAI), 2023

    2023

  38. [38]

    J. Guo, W. Bao, J. Wang, Y . Ma, X. Gao, G. Xiao, A. Liu, J. Dong, X. Liu, W. Wu, A comprehensive evaluation framework for deep model robustness, Pattern Recogni- tion 137 (2023) 109308. 21

    2023