{"id":"af19c8ec-b28a-40e2-8299-1ec1ca48e507","arxiv_id":"2607.19855","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A greedy, query-budgeted attack-ensemble framework approximates the minimum-perturbation frontier and yields an ε-free defense ranking (DOI).","lead":"This paper proposes replacing fixed-budget adversarial-robustness benchmarks with curve-based evaluation: a pool of minimum-norm attacks is budgeted greedily to approximate the strongest attack frontier across ℓ0, ℓ1, ℓ2, and ℓ∞ perturbations. A new Defense Optimality Index ranks models by their gap to the best observed robustness, avoiding the choice of a single perturbation size ε.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"AOI is normalized by the pool's own frontier; with only white-box gradient attacks in the pool, 98–99.9% AOI and 'match/exceed AA' can reflect a weak reference rather than true worst-case robustness.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the empirical attack frontier is defined relative to a hand-picked, white-box-only attack pool. I sharpen this by showing the specific mechanism—the AOI normalization in Eq. (5) divides by the pool frontier's area gap, so a weak pool mechanically raises all AOI values toward 1. This makes the reported 98–99.9% numbers and the 'match or exceed AutoAttack' claim difficult to interpret as evidence of closeness to true worst-case performance. The concern is explicitly acknowledged in Sec. 7, but the paper's abstract and conclusion nonetheless present the benchmark as a practical replacement for fixed-ε leaderboards, which is only justified if the pool is comprehensive. Other issues raised by the reader—coarse AA curve reconstruction, missing error bars, malformed code URL—are real but affect precision and reproducibility, not the conceptual validity of the framework. The concrete test I propose directly probes the load-bearing assumption: adding a strong black-box attack to the pool would reveal whether the frontier, AOI, and DOI rankings are stable or pool-dependent. If the frontier shifts, the central benchmark claim must be conditioned on pool coverage; if not, the concern is resolved. Therefore the correct verdict remains CONDITIONAL, as the reader already concluded, with no change needed.","tokens_in":23318,"tokens_out":8028,"duration_ms":81825,"concrete_test":"Add a strong withheld attack family—e.g., a minimum-norm adaptation of AutoAttack's Square attack or HopSkipJump—to the pool, keeping all other settings fixed. Recompute (i) the full-pool frontier and AUREC_a* on the 17 training models, (ii) the AOI of E12k on the 13 held-out defenses, and (iii) the DOI rankings of all 30 defenses. If AUREC_a* decreases materially (e.g., >1%) or any DOI rank shifts by more than 2 positions, the reported AOI/DOI are artifacts of the incomplete pool and the benchmark claims must be conditioned on pool coverage. If nothing changes, the concern does not land.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claims are pool-relative in a way that can inflate every optimality number. In Eq. (5), AOI(a) = (ρ0·εmin − AUREC_a) / (ρ0·εmin − AUREC_a*), where a* is the per-sample minimum over the fixed attack pool. If the pool is weak, AUREC_a* is small (the frontier curve sits high), so the denominator is small and all attacks—including the paper's ensembles and even AutoAttack—receive AOI values near 1. Table 4's AA AOI values of 96–99.9% are consistent with this compression. Thus the headline that E12k 'matches or exceeds AutoAttack' and that DOI gives an ε-independent ranking presupposes that the pool's frontier approximates the true worst-case curve. Section 7 explicitly concedes that the pool contains only white-box, gradient-based attacks, so defenses using gradient obfuscation, black-box-only access, or patch/semantic threat models are measured against a frontier that may be far above the true minimal perturbations. Since DOI (Eq. 12) is a ratio of AUREC to the defense-frontier AUREC, pool weakness can also compress DOI values and reorder defenses. This is the load-bearing assumption: if a strong attack family is missing, the benchmark's optimality and rankings are optimistic and can invert.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that fixed-ε, predefined-ensemble evaluation (RobustBench/AutoAttack) is fundamentally limited: it samples a single point of the robustness–perturbation curve, gives no evidence of closeness to worst-case attacks, and couples evaluation quality to a non-adjustable cost. The authors propose a curve-based framework built on a pool of minimum-norm attacks (PDPGD, PDGD, FMN, ALMA, DDN, σ-zero, plus their new APGD_min). The 'attack frontier' is the per-sample minimum distance over the pool at full budget; ensemble evaluation is formalized as a budget-constrained frontier-approximation problem (Eq. 8), solved by a greedy query allocation (Algorithm 1) that produces nested ensembles E_4k ⊆ E_8k ⊆ E_12k maximizing the pool-relative Attack Optimality Index. The allocation is selected on 17 training defenses and evaluated on 13 held-out defenses on CIFAR-10/ImageNet across ℓ0–ℓ∞. They report that the ensembles match or exceed AutoAttack's AOI on most held-out models and, at the reference ε, on most ℓ2 (13/13) and ℓ1 (10/13 at 12k) benchmarks, but not ℓ∞ (5/13). A Defense Optimality Index (DOI, Eq. 12) ranks defenses by the ratio of their AUREC to the defense-frontier area and is claimed to be ε-independent and stable across budget tiers.","tokens_in":23638,"tokens_out":28616,"duration_ms":282590,"significance":"Strengths include a coherent formalization, a genuine held-out model split with a narrow generalization gap (Figure 2), nested budget tiers with a monotonic tightening property, a sound early-stopping rule (§5.3), and released code. The demonstration that fixed-ε rankings are unstable (Figure 5) usefully instantiates a known but under-documented concern. If the empirical claims survive a fair AutoAttack baseline, this would be a valuable budget-aware, curve-based evaluation tool. The stress-test concern largely lands, however: the 98–99.9% AOI values and 'match/exceed AutoAttack' statements are relative to an empirical pool that is restricted to white-box, gradient-based attacks (Table 2; §7). Because the AOI normalization (Eq. 5) compresses toward 1 when the pool is weak or saturated, the high values are consistent with a self-referential reference rather than proximity to the true worst case. The held-out split mitigates the overfitting half of the AOI-as-objective-and-metric circularity, but not the pool-relativity. These issues are fixable within the manuscript's scope; they require re-scoping the claims, a fairer AA baseline, and uncertainty quantification.","major_comments":[{"comment":"The AA baseline in the AOI comparison is reconstructed by evaluating AA at ten ε values (nine distance percentiles from the authors' own frontier plus ε_ref) and linearly interpolating. Since robustness curves are step functions, this is an uncontrolled approximation whose error can match the claimed effects: ℓ∞ held-out AOI differences vs AA are ~0.1–2 pp (e.g., C3: 99.62 vs 99.75). Because the grid comes from the ensemble's own distances, the comparison is not neutral. The ℓ∞ discrepancy between the two tables sharpens this: at 12k the ensemble matches/exceeds AA's AOI on 10/13 held-out models, but at the reference ε only on 5/13, with the mean difference in AA's favor (−0.2±0.3 pp). The abstract's 'match or exceed AutoAttack on most defenses at every budget tier' is unsupported for ℓ∞. Please use a minimum-norm AA baseline (binary search for per-sample distances) or bound the reconstr","section":"§5.1, Tables 4 and 6"},{"comment":"AOI is normalized by the gap between the trivial rectangle and the pool's own frontier. A weak or saturated pool shrinks that denominator and compresses every AOI toward 1; Table 4's 96–99.9% values, including AA's, are consistent with this compression. The pool (Table 2) contains only white-box gradient attacks, including the authors' own APGD_min, and §7 concedes gradient-obfuscating or non-differentiable defenses are 'not faithfully evaluated.' The DOI (Eq. 12) is normalized by a defense frontier built from the same pool, so a missing attack family inflates the DOI of exactly the defenses that resist the pool and can reorder Table 7. The held-out split answers the overfitting half of the AOI self-referentiality, but not this pool-relativity. Please qualify the claims as pool-relative throughout (including 'comprehensive pool'/'worst-case' in the abstract) and add a sensitivity check w","section":"§2.2 (Eq. 5), §7"},{"comment":"The cost comparison is asymmetric. Table 6's Q_AA is the query cost of a single AA evaluation at the reference ε (up to 7,566 queries on ℓ2), but the AOI comparison in Table 4 requires AA to be run at ten ε values (the nine percentiles plus ε_ref) to reconstruct a curve; the total cost of that curve is never reported. The claim of 'fixed and controllable query cost' versus AA's 'variable' cost is therefore not apples-to-apples. Please report the total query cost incurred by each side to produce the reported curve-based estimates, and state explicitly what Q_AA covers.","section":"§5.1–5.2, Table 6"},{"comment":"Results are on a fixed 1,000-sample subset per model–norm pair, yet Table 7 reports DOI rank differences of a few tenths of a percentage point (e.g., ℓ0 C15 vs C8 at 4k, 60.3 vs 59.8) and asserts rank stability with no uncertainty quantification. With N=1,000, the standard error of a robust-accuracy estimate at p≈0.5 is ~1.5 pp, and this noise propagates into AUREC, AOI, and DOI. Without bootstrap confidence intervals over samples, it is unclear whether the reported 0.1–0.5 pp separations in DOI and the per-model AOI comparisons are signal or noise. Please add confidence intervals or a paired per-sample test for the ensemble-vs-AA comparisons and justify the sample size.","section":"§5.1, Tables 5 and 7"}],"minor_comments":[{"comment":"The indented Restarts/Targeted rows report cumulative queries and en-dashes mean inactive groups, but the caption does not explain how group totals relate to the base run; add a worked example. The URL in the caption, 'https://github.com/adversarial-frontier.github.io', appears malformed — it should point to an actual repository (and to the anonymized copy for review).","section":"Table 3 and captions"},{"comment":"'All baseline solvers are executed with their default hyperparameters, The only exception is...' — comma splice; also the 'closed-form ℓ2/3 concave surrogate proximal operator' used for PDPGD in ℓ0 is introduced without a citation. Please cite the surrogate or give its definition.","section":"§5.1"},{"comment":"The reference list contains formatting artifacts (e.g., [1] 'V ol.', [4] missing venue formatting, [5] broken line breaks around the DOI). A cleanup pass is needed.","section":"References"},{"comment":"The term 'attack frontier' is used both for the per-sample minimizer a* and for the frontier curve ρ_a*(ε); the two senses appear together in Eqs. (3)–(5) and in Figure 2's caption. Please use distinct terms or define both explicitly at first use.","section":"§2.2, §3"},{"comment":"The sentence claiming that black-box, semantic, or patch-based perturbations can be added 'at the sole cost of generalizing the query unit' is unsupported: semantic perturbation families have no natural query-based cost accounting. This should be framed explicitly as future work rather than as a property of the framework.","section":"§7"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is publishable in principle, but the headline empirical claim needs to be reconciled with the evidence. The most important revision is the AutoAttack baseline: the current Table 4 comparison uses a reconstructed AA curve that is not a faithful representation of AA's minimum-norm behavior, and the ℓ∞ discrepancy between Table 4 and Table 6 suggests the AOI-based 'match or exceed' claim may be an artifact. I also recommend requiring the authors to state pool-relativity in the abstract and contributions, and to add uncertainty quantification, since many reported differences are within sampling noise. The paper's limitations section is exemplary, and the extensibility argument (adding attack families later) is reasonable. If these points are addressed, I would support publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper deserves referee time, but read the optimality numbers with a grain of salt. The genuinely new piece is the greedy query-budget allocation that builds minimum-norm attack ensembles over ℓ0/ℓ1/ℓ2/ℓ∞, with nested 4k/8k/12k tiers and a held-out split of 17/13 defenses. The generalization result — train vs. held-out AOI gap below 0.4 points — is real evidence, and APGD_min plus the DOI/defense-frontier ranking are useful additions. The paper is also honest in Section 7 about the main caveat: the pool is white-box gradient-based only.\n\nThe soft spot is exactly the pool-relative normalization. AOI is defined relative to the best attack in the pool; if the pool is weak, the denominator shrinks and every attack's AOI gets squeezed toward 1. Table 4 shows this: AA itself scores 96–99.9% AOI on every norm. So the headline 'match or exceed AutoAttack' is comparing two things that are both close to the same self-defined target, and the margins are small enough to be noise. The more informative comparison is robust accuracy at the reference ε (Table 6), and there the story is more mixed: 28/39 held-out model–norm pairs, with only 5/13 on ℓ∞. The abstract's 'most defenses' hides that.\n\nThere are smaller issues: AA's curve is reconstructed from 10 linearly interpolated points on a grid taken from the authors' own frontier, which biases the AOI comparison; reported numbers have no error bars over the 1,000-sample subsets; and the code link in the paper is not usable (malformed URL, no commit hash). None of these are fatal.\n\nThe framework stands as a query-controlled, curve-based alternative to fixed-ε evaluation, with the important qualifier that it measures closeness to the empirical frontier of a fixed attack pool, not to true worst-case robustness. DOI rankings are likewise relative to the specific defense pool.\n\nWho is this for? Researchers building or using robustness benchmarks — particularly anyone working on query-budget-aware evaluation or extending RobustBench-style leaderboards. It deserves a serious referee: the idea is coherent, the held-out validation is above average, and the limitations are disclosed rather than buried. I'd send it to review, but I'd also ask the authors to fix the code link, report variance, and soften the abstract so it doesn't imply absolute optimality.","headline":"Solid, pool-relative benchmark paper — worth refereeing, but the near-100% AOI scores are less meaningful than they look.","tokens_in":24215,"tokens_out":3610,"would_cite":false,"duration_ms":37415,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Robustness should be evaluated as a full curve, not a single epsilon; budgeted minimum-norm attack ensembles approximate the worst-case frontier to 98–99.9%, and a curve-area index ranks defenses epsilon-free.","keywords":["adversarial robustness evaluation","robustness–perturbation curves","minimum-norm attacks","attack ensembles","query budget allocation","attack optimality index","defense optimality index","epsilon-free ranking"],"falsifier":"Take any defended model and any sample from the evaluation subset, run the full 12k-query pool, and record the per-sample minimum distance; then run a strong attack of a class absent from the pool, for example a query-efficient black-box attack or one designed for gradient obfuscation. If that outside-pool attack finds a misclassifying perturbation strictly smaller than the pool's minimum on even one sample, then the empirical frontier is not tight and every AOI/DOI value derived from it is an overestimate.","tokens_in":23165,"feed_emoji":"🛡️","tokens_out":11560,"duration_ms":104209,"temperature":0.7,"pith_summary":"The paper tries to establish that fixed-budget, single-point robustness evaluation is incomplete: curves for different models can cross, so rankings at one epsilon can flip at another. It replaces the question \"how robust is this model at ε\" with \"how close to the per-sample worst case can an attack pool get under a query budget,\" formalizing the attack frontier and a greedy budget-allocation algorithm whose ensembles reach 98–99.9% of that frontier on held-out models across ℓ0, ℓ1, ℓ2, and ℓ∞. On most held-out defenses, the ensembles match or exceed the standard fixed-budget attack ensemble at a controllable and often smaller query cost. The companion Defense Optimality Index aggregates each model's gap to the best curve in a defense pool, giving an epsilon-free ranking that is stable across budget tiers. A sympathetic reader would care because it offers practitioners a principled trade-off between evaluation cost and tightness, and a way to compare defenses without choosing an operating point.","feed_headline":"~99% of the worst-case frontier reached by budgeted attack ensembles","feed_subtitle":"A query-controlled attack ensemble matches the fixed-budget baseline on most defenses; new index ranks them epsilon-free.","key_machinery":"The carrying object is the empirical attack frontier: the per-sample minimum of distances returned by a pool of minimum-norm attacks, whose induced curve is the tightest robustness estimate the pool can give. On top of it sit two area-based indices — the Attack Optimality Index (normalized area gap of one attack's curve to the frontier) and the Defense Optimality Index (ratio of a model's curve area to the maximum over models) — and the greedy budget-allocation algorithm that assigns query increments to attacks by AOI gain per added query, producing a nested, monotonically tightening family of ensembles. The monotonicity relies on the fact that each attack's distance is an upper bound on the","core_discovery":"The central claim is that the empirical attack frontier — the per-sample minimum perturbation found by any attack in a pool — can be approximated almost exactly by a small, adaptively budgeted subset of that pool, and that the residual gap is a measurable, budget-controllable quantity. Experiments show the greedy ensembles reach an attack-optimality index of 98.1–99.8% on ℓ0 and 98.3–99.9% on ℓ1–ℓ∞ held-out defenses, improving monotonically with query budget; the ℓ2 ensemble matches or exceeds the fixed-budget standard on all 13 held-out models already at 4k queries, and the ℓ1 ensemble does so on all 13 at 12k. The paper further claims that robustness–perturbation curves can cross at budget","pith_inferences":["Inference — Because every index is pool-relative, extending the pool with black-box, patch-based, or semantic attacks would likely change both the AOI numbers and the DOI rankings; the paper's headline comparisons with the fixed-budget ensemble are against one particular pool, and a broader pool could erode some of the reported margins.","Inference — The same min-composition machinery could sandwich the true robustness curve between the empirical frontier (an upper bound on minimal distance) and a certified lower bound, turning the \"gap to worst case\" from a heuristic into an interval with guarantees; the paper does not propose this.","Inference — The epsilon-free perspective might push adversarial training to optimize area under the curve rather than accuracy at a single adversarial budget, changing which defenses are considered state of the art; this is a consequence of switching from point evaluation to curve evaluation, not something the paper tests."],"forward_implications":["Rankings at a single ε are unreliable: crossing curves mean the leaderboard order can flip at budgets beyond the reference; the paper observes 17 CIFAR-10 defenses moving at least 5 ranks on ℓ2 across the evaluated grid.","Evaluation quality becomes a user-chosen budget rather than a fixed pipeline: increasing the query budget from 4k to 12k monotonically raises ensemble AOI, with the largest gain in ℓ1 (+1.12) and the smallest in ℓ∞ (+0.28).","The greedy allocations generalize: held-out AOI stays within about 0.4 percentage points of the training models, so ensembles tuned on one model set can evaluate unseen defenses.","The ℓ0 norm, previously without a benchmark baseline, can now be evaluated: held-out AOI reaches 98.1–99.8% at 12k queries.","A candidate defense can be rejected after the cheapest tier if its 4k curve already lies below the incumbent's 12k curve, because additional queries can only reveal it to be weaker."],"fun_headline_variants":["Budgeted attack ensembles hit 99% of worst-case frontier","Query-budgeted attacks match AutoAttack on most defenses","New index ranks defenses without picking a perturbation budget","Robustness curves cross; budgeted ensembles still win","Attack frontier approximated by small adaptive attack subsets"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the hand-picked pool of white-box, gradient-based minimum-norm attacks finds the true per-sample minimal perturbation; if the strongest attack family for a given defense is missing — black-box, gradient-obfuscation, patch-based, or semantic — then the frontier, the 98–99.9% optimality numbers, the comparison with the fixed-budget ensemble, and the DOI rankings are all optimistic and could invert. The paper's own Section 7 acknowledges this lim","fun_headline_variants_meta":{"raw":{"variants":["Budgeted attack ensembles hit 99% of worst-case frontier","Query-budgeted attacks match AutoAttack on most defenses","New index ranks defenses without picking a perturbation budget","Robustness curves cross; budgeted ensembles still win","Attack frontier approximated by small adaptive attack subsets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000166,"raw_usage":{"total_tokens":1146,"prompt_tokens":855,"completion_tokens":291,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":599,"completion_tokens_details":{"reasoning_tokens":214}},"tokens_in":599,"tokens_out":291,"duration_ms":3726,"temperature":1.0,"reasoning_tokens":214,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T11:31:15.623301+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take any defended model and any sample from the evaluation subset, run the full 12k-query pool, and record the per-sample minimum distance; then run a strong attack of a class absent from the pool, for example a query-efficient black-box attack or one designed for gradient obfuscation. If that outside-pool attack finds a misclassifying perturbation strictly smaller than the pool's minimum on even one sample, then the empirical frontier is not tight and every AOI/DOI value derived from it is an overestimate.","supporting_citations":[],"review_version":1}