Pith. sign in

REVIEW 3 major objections 5 minor 48 references

Large-scale Testing Global Optimization Methods with Black-box Adversarial Attacks

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Black-box adversarial attacks can serve as realistic high-dimensional benchmarks for global optimization, supported here by evidence of multimodality and a seven-optimizer comparison.

desk verdict Promising benchmark idea, but the multimodality evidence is a threshold-crossing artifact rather than proof of multiple local optima. read the letter →

arxiv 2608.13296 v1 pith:AF2N2F5V submitted 2026-08-13 cs.LG cs.AI

Wojciech Zarzecki , Jaros{l}aw Arabas This is my paper · ORCID
classification cs.LGcs.AI
keywords black-boxadversarialattackglobaloptimizationbenchmarkmultimodalityevolutionaryalgorithmsdifferentialevolutionmetaheuristicsCIFAR-10ImageNet
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes that black-box adversarial attacks—finding a barely visible perturbation to an image that flips a classifier's decision using only the classifier's output probabilities—should be treated as benchmark problems for global optimization. The authors formulate the attack as minimizing $L(\delta) = -\log p_{c(x)}(x+\delta) - \alpha\|\delta\|_2^2$ over the feasible hypercube $[-\varepsilon,\varepsilon]^n$, an objective with thousands to hundreds of thousands of dimensions. They show that a stochastic greedy local search converges to different perturbations from different random seeds, which they read as evidence that the objective is multimodal. They then compare seven population-based optimizers on CIFAR-10 and ImageNet, reporting that GEN and SHADE find successful low-strength attacks most reliably. If the benchmark idea holds, global optimization methods can be tested on realistic, large-scale problems tied to modern machine learning rather than small analytical functions from the 1970s.

What carries the argument

The load-bearing object is the adversarial attack objective $L(\delta)$ from equation (1), a bound-constrained minimization over the hypercube $[-\varepsilon,\varepsilon]^n$. The central mechanism for evaluating multimodality is the SGA-BSR local search, a stochastic pixel-increment search with binary-search refinement; its seed-dependent convergence is the paper's evidence for multiple local optima. The benchmark protocol then applies this objective, with a fixed population size and query budget, to per-image attack problems so that optimizers can be compared on success rate, queries to first success, and normalized disturbance strength.

What would settle it

Take a provably unimodal objective over the same hypercube, such as a shifted quadratic with the same clipping, and run the same SGA-BSR procedure from many seeds; if the final perturbations still differ noticeably across seeds, then seed-dependent convergence does not establish multimodality, and the benchmark claim would need a different empirical basis.

Watch

Extended reading notes

Core claim

The central claim is that the black-box adversarial attack problem can be used as a large-scale global optimization benchmark. The paper defines the attack as a minimization of $L(\delta) = -\log p_{c(x)}(x+\delta) - \alpha\|\delta\|_2^2$ subject to $\delta \in [-\varepsilon,\varepsilon]^n$, where $p_j$ is the classifier's output probability for class $j$ and $c(x)$ is the true label. Attack difficulty is controlled by $\varepsilon$, the maximum per-pixel perturbation; on CIFAR-10 the search space has 3,072 dimensions and on ImageNet 150,528 dimensions. To support the benchmark claim, the authors design a local search method, SGA-BSR, that greedily grows a perturbation one pixel at a time and then shrinks each altered pixel by binary search while preserving misclassification. Running this local search from many random seeds yields different final perturbations and different objective values, which the paper takes as evidence of multiple local optima and hence multimodality. A comparison of seven metaheuristics shows that GEN (a genetic algorithm) and SHADE (an adaptive differential evolution variant) most often achieve high attack success with the smallest normalized perturbation strength, while GWO struggles without regularization and INFO behaves like a cheap local search.

Load-bearing premise

The argument that the attack objective is multimodal rests on the premise that repeated runs of a random pixel-growing search would all converge to the same perturbation if the objective had only one minimum; this premise can fail on flat or ridge-shaped unimodal functions, so the evidence is not conclusive on its own.

Editorial extensions

If this is right

  • If the benchmark proposal is accepted, global optimization methods can be stress-tested on problems with 3,072 to 150,528 continuous variables, far beyond the dimension range of standard analytical suites.
  • The same objective supplies a natural query-cost metric: every evaluation is a full forward pass of a neural network, so a method's efficiency can be measured in real classifier queries rather than abstract function calls.
  • The reported ranking—GEN and SHADE achieving high success rates with the smallest perturbation strength—gives future work two concrete baselines for score-based adversarial attacks.
  • Because attack success is strongly controlled by the perturbation bound $\varepsilon$ and moderately helped by a moderate regularization weight, benchmark designers can tune difficulty simply by adjusting these two parameters.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An extension the authors do not pursue: the same benchmark could rank methods by query efficiency under a hard query budget, since each evaluation costs a full forward pass.
  • If the benchmark is adopted, the difficulty gradient observed between CIFAR-10 and ImageNet suggests that input resolution and label-set size could be used to generate problem instances of controlled difficulty.
  • A cautionary test: rerunning SGA-BSR on a provably unimodal surrogate with flat regions would show whether seed-dependent convergence alone is enough to infer multimodality.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes that black-box adversarial attack (BBAA) problems, defined by maximizing the objective in Eq. (1) over the feasible hypercube [-epsilon, epsilon]^n, constitute a realistic, high-dimensional benchmark for global optimization methods. The authors introduce a stochastic local search method (SGA-BSR) and use seed-dependent outcomes of this method to claim that the BBAA objective is multimodal. They then compare seven metaheuristics (DE, GEN, GWO, INFO, JADE, SADE, SHADE) on CIFAR-10 and ImageNet classifiers using nominal mealpy defaults, reporting success rates, query counts, perturbation strengths, and objective values. The central claims are that BBAA is a demanding global optimization task and that GEN and SHADE are the most efficient of the tested methods.

Significance. If the multimodality claim were established, the paper would provide a valuable new class of benchmark problems that connects global optimization to modern machine learning, with the practical advantage of being defined by real classifiers rather than synthetic functions. The paper's strengths include the release of a framework for testing new optimizers, the use of a standard attack formulation, and the absence of fitted parameters: all optimizers run with library defaults. However, the load-bearing evidence for multimodality is not sufficient, and the sign convention of the objective is inconsistent. The empirical comparison also lacks the statistical rigor needed to support the claimed ranking. These issues are fixable, but they currently prevent acceptance.

major comments (3)
  1. [Section 4.2, Tables 1 and 2] Seed-dependent SGA-BSR endpoints do not establish multimodality. The growth phase of Algorithm 1 stops as soon as the classifier's label flips; this is not convergence to a local optimum of L but rather a first crossing of the decision boundary, i.e., a level-set crossing. On a strictly unimodal objective, independent stochastic coordinate-ascent paths can cross that level set at different points, and the subsequent binary-search refinement will then produce different final perturbations. Therefore the premise stated in Section 3.2, that on a strictly unimodal problem attacks from different seeds would converge to the exact same optimal perturbation, is false in general. The observed nonzero standard deviations in Figures 2 and 3 are exactly what one would expect from a threshold-crossing stochastic search on a unimodal landscape. To support the central benchmark claim, the authors need a control experiment on a known unimodal landscape, or direct evidence of multiple local optima (for example, by a systematic restart analysis on a reduced-dimensional surrogate or by verifying that different SGA-BSR endpoints are local maxima of L in the sense of all coordinate-wise moves). Without this, the paper's main conclusion that BBAA is multimodal and therefore a suitable global-optimization benchmark is not established.
  2. [Section 4.2, Tables 1 and 2] The sign convention of the objective is internally inconsistent. Eq. (1) is introduced as a bound-constrained minimization problem for L(delta) = -log p_c(x+delta) - alpha ||delta||_2^2, but Algorithm 1 accepts a perturbation when L(delta_tmp) >= L(delta), i.e., it maximizes L, and Tables 1 and 2 mark the objective column with an up arrow, indicating that larger values are better. If minimization is the intended formulation, the algorithm and tables should accept and report smaller L values; if maximization is intended, Eq. (1) and the surrounding text should be rewritten accordingly. This inconsistency affects the interpretation of the reported objective values, the role of the regularization term, and the comparability of methods across tables, so it must be resolved before the experimental results can be interpreted.
  3. [Section 4.2, Tables 1 and 2] The ranking claim that 'GEN and SHADE dominate' is not supported by statistical analysis. The reported quantities are means and standard deviations aggregated over images and runs, but the experimental design performs 100 runs per image and therefore has a hierarchical structure: variation across images and variation across runs are conflated in the pooled standard deviations. The paper does not report paired comparisons across the same images, confidence intervals, or significance tests. For example, in Table 1 for alpha=0.1 and epsilon=0.2, GEN has success rate 97.4% and SHADE 89.6%, while the objective values are 2.63 +/- 1.60 and 1.93 +/- 1.81, respectively; without per-image paired analysis it is not clear that these differences are robust. The authors should either provide per-image results or use appropriate statistical tests before claiming that GEN and SHADE are the most efficient methods.
minor comments (5)
  1. [Section 3.2, Figures 2 and 3] The number of SGA-BSR runs is inconsistent: the text states 1000 different random seeds, Figure 2's caption says 100 independent runs, and Figure 3's caption mentions 1000 runs while also saying 100 random seeds. Please clarify the actual number of runs and ensure all captions and text agree.
  2. [Section 1.3] The phrase 'proving a framework' should be 'providing a framework'; also, the sentence would benefit from a reference to the anonymous archive where the framework is made available.
  3. [Section 2, Eq. (1)] The notation p_c(x) in Eq. (1) is ambiguous: since c(x) is the ground-truth class, it would be clearer to write p_{c(x)}(x+delta) to make explicit that the probability is evaluated for the ground-truth class.
  4. [Section 4.1] The sentence 'Each run of the optimization process is terminated after exceeding the admissible limit of perturbations' is unclear: it should state which budget is exceeded (e.g., the maximum number of iterations or the query limit) and how a run is classified as unsuccessful.
  5. [General] There are several typographical and stylistic issues, including 'Costa et. al' (should be 'Costa et al.'), 'inrease' (should be 'increase'), and the inconsistent use of 'eg.' and 'cf.'; a careful proofreading pass is recommended.

Circularity Check

1 steps flagged · score 4.0 of 10

Multimodality evidence is self-referential: SGA-BSR's random first-crossing stopping rule guarantees seed-dependent endpoints, which are then read as proof of multiple local optima.

  1. self definitional [Section 3.2, Figure 2 caption and preceding paragraph; Algorithm 1 lines 6 and 12-13]
    "This is evidence that different seeds yield different successful perturbation vectors; therefore, the attack problem is multimodal."

    The paper equates 'unimodal' with 'all SGA-BSR runs return the same optimal perturbation.' But Algorithm 1 terminates at the first misclassification (line 12-13) after a random coordinate increment (line 6), so different seeds are almost surely different random first-crossing points on the decision boundary even if the objective L is strictly unimodal. The observed diversity of endpoints is therefore a by-construction property of the stochastic search rule, not an independent measurement of the landscape. The paper's own stated aim, 'We formulate the local search method to demonstrate that the optimization problem is multimodal' (Section 1.3), makes the demonstration self-referential: the algorithm's built-in randomness supplies the diversity that is then attributed to the objective.

full rationale

The paper's only candidate circularity is the multimodality argument in Section 3.2. The paper asserts that a unimodal objective would make all SGA-BSR runs converge to the same optimal perturbation, then uses seed-dependent endpoints as proof of multimodality. As operationalized, this is circular: Algorithm 1 stops at the first misclassification after random coordinate growth, so the endpoint is a random boundary-crossing point rather than a local optimum of L. Different seeds therefore produce different perturbations by construction, even on a strictly unimodal landscape, so the evidence is generated by the algorithm rather than measured from the objective. The phrase 'We formulate the local search method to demonstrate that the optimization problem is multimodal' makes the self-reference explicit. However, this is a single supporting inference. The efficiency comparison of GEN, SHADE, DE, GWO, INFO, JADE, and SADE on realistic attack problems, the use of a standard objective and default optimizer settings, and the high-dimensionality observations are independent content. There are no fitted parameters renamed as predictions and no load-bearing self-citations. The score of 4 reflects one central evidence step that reduces by construction while the remaining empirical contributions stand independently.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The benchmark objective rests on two hand-set hyperparameters (alpha and epsilon) and an unreported step size for the multimodality probe. The multimodality claim additionally relies on an implicit convergence assumption for stochastic coordinate ascent that is not stated as a limitation and is likely false. No new physical or mathematical entities are postulated.

free parameters (3)
  • regularization weight alpha = values 0, 0.1, 1, 10, 100
    Controls the trade-off between the log-probability term and the squared L2 perturbation penalty in objective Eq. (1). It is chosen by hand and changes benchmark difficulty, but is not fitted to data.
  • perturbation bound epsilon = values 0.01, 0.1, 0.2
    Defines the feasible hypercube [-epsilon, epsilon]^n; selected by hand and strongly determines success rates.
  • SGA-BSR step size eta = not reported in the manuscript
    Algorithm 1 increments a randomly chosen pixel by eta; the value is needed to reproduce the multimodality probe but is omitted from the text.
assumptions (4)
  • domain assumption A nonlinear classifier induces a multimodal objective (1).
    Section 3 opening states this expectation, but nonlinearity alone does not guarantee multimodality; it is a heuristic motivation.
  • ad hoc to paper On a unimodal objective, independent runs of SGA-BSR would converge to the same optimum.
    Section 3.2 uses this premise to turn seed variance into evidence of multimodality. Stochastic coordinate ascent can stall on plateaus or ridges and produce different endpoints on a unimodal function, so this premise is load-bearing and unjustified.
  • domain assumption Default hyperparameters from the mealpy library are a fair basis for comparing optimizers.
    Section 4.1 states all optimizers use defaults; some methods may be advantaged or disadvantaged, so rankings depend on this choice.
  • domain assumption Score-based probability access is the relevant black-box attack setting.
    Section 1.3 restricts BBAA to probability feedback, excluding decision-only attacks; the benchmark value is scoped to score-based attacks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Large-scale Testing Global Optimization Methods with Black-box Adversarial Attacks." pith.science (2026). https://pith.science/paper/AF2N2F5V

@misc{pith2026260813296,
  author       = {Pith},
  title        = {Pith review of: Large-scale Testing Global Optimization Methods with Black-box Adversarial Attacks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AF2N2F5V}},
  note         = {Machine review of arXiv:2608.13296}
}
read the original abstract

Existing global optimization benchmark suites are of a moderate size and are based on a small number of analytical functions that date back even to the 1970s. This causes a risk of biasing the development of global optimization methods. We argue that the tasks related to the black-box adversarial attack (BBAA) can serve as valuable global optimization benchmark in many-dimensional space. We demonstrate the efficiency of several types of evolutionary algorithms and other metaheuristics in solving example BBAA problems. Thus, we take a step towards convergence of global optimization methods to the challenges and needs that arise in the modern machine learning field.

Figures

Figures reproduced from arXiv: 2608.13296 by the authors.

Figure 1
Figure 1. Example adversarial attack: an image that is originally classified as a [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Pixel-wise mean and standard deviation of the perturbation [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Per-image variance of SGA-BSR solutions across 1000 indepen￾dent runs (α = 0.1). For each image, Algorithm 1 is run with 100 random seeds. Boxplots for objective function value: top two panels: CIFAR-10, bottom: ImageNet. Columns are images, boxes are IQR over seeds, whiskers 1.5 IQR, dots outliers. disturbance strength defined as strength(δ) = ||δ||2 √ n (2) where n is the vector length. All optimizers were run wit… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Example image from ImageNet with two successful attack examples for [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 33 canonical work pages

  1. [1]

    In: 27th USENIX security symposium (USENIX Security 18)

    Adi, Y., Baum, C., Cisse, M., Pinkas, B., Keshet, J.: Turning your weakness into a strength: Watermarking deep neural networks by backdooring. In: 27th USENIX security symposium (USENIX Security 18). pp. 1615–1631 (2018)

    work page 2018

  2. [2]

    In: Proceedings of the genetic and evolutionary computation conference

    Alzantot, M., Sharma, Y., Chakraborty, S., Zhang, H., Hsieh, C.J., Srivastava, M.B.: Genattack: Practical black-box attacks with gradient-free optimization. In: Proceedings of the genetic and evolutionary computation conference. pp. 1111– 1119 (2019)

    work page 2019

  3. [3]

    In: European conference on computer vision

    Andriushchenko, M., Croce, F., Flammarion, N., Hein, M.: Square attack: a query- efficient black-box adversarial attack via random search. In: European conference on computer vision. pp. 484–501. Springer (2020)

    2020

  4. [4]

    Brendel, W., Rauber, J., Bethge, M.: Decision-based adversarial attacks: Reliable attacks against black-box machine learning models (2017)

    work page 2017

  5. [5]

    2017 IEEE Symposium on Security and Privacy (SP) pp

    Carlini, N., Wagner, D.A.: Towards evaluating the robustness of neural networks. 2017 IEEE Symposium on Security and Privacy (SP) pp. 39–57 (2016),https: //api.semanticscholar.org/CorpusID:2893830

    work page 2016

  6. [6]

    In: 2020 ieee symposium on security and privacy (sp)

    Chen, J., Jordan, M.I., Wainwright, M.J.: Hopskipjumpattack: A query-efficient decision-based attack. In: 2020 ieee symposium on security and privacy (sp). pp. 1277–1294. IEEE (2020)

    2020

  7. [7]

    Chen, P.Y., Zhang, H., Sharma, Y., Yi, J., Hsieh, C.J.: Zoo: Zeroth order optimiza- tion based black-box attacks to deep neural networks without training substitute models (2017),https://api.semanticscholar.org/CorpusID:2179389

    work page 2017

  8. [8]

    Chen, Q., Liu, B., Zhang, Q., Liang, J., Suganthan, P., Qu, B.: Problem definitions and evaluation criteria for CEC’2015 special session on bound constrained single- objective computationally expensive numerical optimization (2014)

    work page 2014

Show all 48 references
  1. [9]

    Choi, J.S., Lee, K., Jeong, J., Xie, S., Shin, J., Lee, K.: Diffusionguard: A robust defense against malicious diffusion-based image editing (2024)

    2024

  2. [10]

    In: International Conference on the Applications of Evolutionary Computation (Part of EvoStar)

    Clare, L., Marques, A., Correia, J.: A comparative analysis of evolutionary ad- versarial one-pixel attacks. In: International Conference on the Applications of Evolutionary Computation (Part of EvoStar). pp. 147–162. Springer (2024)

    2024

  3. [11]

    IEEE Access12, 61113–61136 (2024)

    Costa, J.C., Roxo, T., Proença, H., Inacio, P.R.M.: How deep learning sees the world: A survey on adversarial attacks & defenses. IEEE Access12, 61113–61136 (2024)

    2024

  4. [12]

    Das, S., Suganthan, P.N.: Problem definitions and evaluation criteria for the CEC’2011 competition on testing evolutionary algorithms on real world problems. Tech. rep. (2010)

    2010

  5. [13]

    (eds.): Towards Global Optimisation 2

    Dixon, L.C.W., Szegő, G.P. (eds.): Towards Global Optimisation 2. North-Holland Publishing Company, Amsterdam (1978)

    1978

  6. [14]

    arXiv preprint arXiv:1903.06396 (2019)

    Elhara, O., Varelas, K., Nguyen, D., Tusar, T., Brockhoff, D., Hansen, N., Auger, A.: COCO: the large scale black-box optimization benchmarking (bbob-largescale) test suite. arXiv preprint arXiv:1903.06396 (2019)

    2019 arXiv

  7. [15]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Eykholt, K., Evtimov, I., Fernandes, E., Li, B., Rahmati, A., Xiao, C., Prakash, A., Kohno, T., Song, D.: Robust physical-world attacks on deep learning visual clas- sification. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1625–1634 (2...

    2018

  8. [16]

    arXiv preprint arXiv:1804.05296 (2018)

    Finlayson, S.G., Chung, H.W., Kohane, I.S., Beam, A.L.: Adversarial attacks against medical deep learning systems. arXiv preprint arXiv:1804.05296 (2018)

    2018 arXiv

  9. [17]

    arXiv preprint arXiv:1412.6572 (2014)

    Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572 (2014)

    2014 arXiv

  10. [18]

    Guo,C.,Gardner,J.R.,You,Y.,Wilson,A.G.,Weinberger,K.Q.:Simpleblack-box adversarial attacks. vol. abs/1905.07121 (2019),https://api.semanticscholar. org/CorpusID:86541092

    2019 arXiv

  11. [19]

    Research Report RR-6869, INRIA (2009),https://inria.hal.science/inria-00369466

    Hansen, N., Finck, S., Ros, R., Auger, A.: Real-Parameter Black-Box Optimiza- tion Benchmarking 2009: Noisy Functions Definitions. Research Report RR-6869, INRIA (2009),https://inria.hal.science/inria-00369466

    2009

  12. [20]

    Hansen, N., Ros, R., Auger, A.: Real-parameter black-box optimization bench- marking 2009: Noiseless functions definitions. Tech. rep. (2009),https://api. semanticscholar.org/CorpusID:270969128

    2009

  13. [21]

    2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) pp

    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) pp. 770–778 (2015),https://api.semanticscholar.org/CorpusID:206594692

    2015

  14. [22]

    In: International Conference on Neural Information Processing

    Ilie, A., Popescu, M., Stefanescu, A.: Evoba: An evolution strategy as a strong baseline for black-box adversarial attacks. In: International Conference on Neural Information Processing. pp. 188–200. Springer (2021)

    2021

  15. [23]

    In: International Conference on Machine Learning (2018),https://api.semanticscholar.org/CorpusID:5046541

    Ilyas, A., Engstrom, L., Athalye, A., Lin, J.: Black-box adversarial attacks with limited queries and information. In: International Conference on Machine Learning (2018),https://api.semanticscholar.org/CorpusID:5046541

    2018

  16. [24]

    Ilyas, A., Engstrom, L., Madry, A.: Prior convictions: Black-box adversarial attacks with bandits and priors (2018)

    2018

  17. [25]

    Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images (2009)

    2009

  18. [26]

    Liang, J.J., Qu, B.Y., Suganthan, P.N.: Problem definitions and evaluation cri- teria for the CEC’2014 special session and competition on single objective real- parameter optimization. Tech. rep., Zhengzhou University and Nanyang Techno- logical University (2013)

    2013

  19. [27]

    Liang, J.J., Qu, B.Y., Suganthan, P.N.: Problem definitions and evaluation criteria for the CEC’2019 competition on single objective real-parameter optimization. Tech. rep., Nanyang Technological University (2018)

    2018

  20. [28]

    Liang, J.J., Qu, B.Y., Suganthan, P.N.: Problem definitions and evaluation criteria for the CEC’2020 competition on single objective real-parameter optimization. Tech. rep., Nanyang Technological University (2020)

    2020

  21. [29]

    Liang, J.J., Qu, B.Y., Suganthan, P.N.: Problem definitions and evaluation criteria for the CEC’2022 competition on single objective real-parameter optimization. Tech. rep., Nanyang Technological University (2022)

    2022

  22. [30]

    Liang, J.J., Qu, B.Y., Suganthan, P.N., Chen, Q.: Problem definitions and eval- uation criteria for the CEC’2017 competition on single objective real-parameter optimization. Tech. rep., Nanyang Technological University (2017)

    2017

  23. [31]

    Liang, J.J., Suganthan, P.N., Deb, K.: Problem definitions and evaluation crite- ria for the CEC’2005 special session on real-parameter optimization. Tech. rep., Nanyang Technological University (2005)

    2005

  24. [32]

    arXiv preprint arXiv:1611.02770 (2016)

    Liu, Y., Chen, X., Liu, C., Song, D.: Delving into transferable adversarial examples and black-box attacks. arXiv preprint arXiv:1611.02770 (2016)

    2016 arXiv

  25. [33]

    arXiv preprint arXiv:1706.06083 (2017) Testing global optimization methods with adversarial attacks

    Madry, A., Makelov, A., Schmidt, L., Tsipras, D., Vladu, A.: Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083 (2017) Testing global optimization methods with adversarial attacks. 17

    2017 arXiv

  26. [34]

    Mallipeddi, R., Suganthan, P.N., Pan, Q., Tasgetiren, M.F.: Problem definitions and evaluation criteria for the CEC’2013 special session on real-parameter opti- mization. Tech. rep., Nanyang Technological University (2013)

    2013

  27. [35]

    Mosli, R., Wright, M., Yuan, B., Pan, Y.: They might not be giants: Crafting black- box adversarial examples with fewer queries using particle swarm optimization (2019)

    2019

  28. [36]

    Omidvar, M.N., Li, X., Tang, K., Mei, Y., Yao, X.: Benchmark functions for the CEC’2012 special session and competition on large scale global optimization. Tech. rep., Nanyang Technological University (2012)

    2012

  29. [37]

    Omidvar, M.N., Li, X., Tang, K., Yao, X.: Benchmark functions for the CEC’2018 competition on large scale global optimization. Tech. rep., Nanyang Technological University (2018)

    2018

  30. [38]

    In: Proceedings of the 2017 ACM on Asia conference on computer and communications security

    Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Prac- tical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia conference on computer and communications security. pp. 506–519 (2017)

    2017

  31. [39]

    Ruhr University Bochum, Institute for Neural Computation: Black-box op- timization competition (bbcomp).https://www.ini.rub.de/PEOPLE/glasmtbl/ projects/bbcomp/index.html

  32. [40]

    International journal of computer vision115(3), 211–252 (2015)

    Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al.: Imagenet large scale visual recog- nition challenge. International journal of computer vision115(3), 211–252 (2015)

    2015

  33. [41]

    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale im- age recognition (2014),https://api.semanticscholar.org/CorpusID:14124313

    2014

  34. [42]

    IEEE Transactions on Evolutionary Computation23(5), 828–841 (2019)

    Su, J., Vargas, D.V., Sakurai, K.: One pixel attack for fooling deep neural networks. IEEE Transactions on Evolutionary Computation23(5), 828–841 (2019)

    2019

  35. [43]

    Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., Fergus, R.: Intriguing properties of neural networks (2013)

    2013

  36. [44]

    In: AAAI Conference on Artificial Intelligence (2018),https://api.semanticscholar.org/CorpusID:44079102

    Tu, C.C., Ting, P.S., Chen, P.Y., Liu, S., Zhang, H., Yi, J., Hsieh, C.J., Cheng, S.M.: Autozoom: Autoencoder-based zeroth order optimization method for at- tacking black-box neural networks. In: AAAI Conference on Artificial Intelligence (2018),https://api.semanticscholar.org...

    2018

  37. [45]

    In: NeurIPS 2020 competition and demonstration track

    Turner, R., Eriksson, D., McCourt, M., Kiili, J., Laaksonen, E., Xu, Z., Guyon, I.: Bayesian optimization is superior to random search for machine learning hy- perparameter tuning: Analysis of the black-box optimization challenge 2020. In: NeurIPS 2020 competition and demonstr...

    2021

  38. [46]

    Journal of Systems Architecture (2023).https: //doi.org/10.1016/j.sysarc.2023.102871

    Van Thieu, N., Mirjalili, S.: Mealpy: An open-source library for latest meta- heuristic algorithms in python. Journal of Systems Architecture (2023).https: //doi.org/10.1016/j.sysarc.2023.102871

    2023

  39. [47]

    2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition pp

    Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition pp. 586–595 (2018),https://api. semanticscholar.org/CorpusID:4766599

    2018

  40. [48]

    Zheng, M., Yan, X., Zhu, Z., Chen, H., Wu, B.: Blackboxbench: A comprehen- sive benchmark of black-box adversarial attacks (2025),https://arxiv.org/abs/ 2312.16979

    2025 arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.