Pith. sign in

REVIEW 3 major objections 2 minor 109 references

Rethinking Evaluation Paradigms in IBP-based Certified Training

T0 review · 3 major / 2 minor · reviewed 2026-06-28 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Pareto-front comparisons via automated multi-objective tuning show prior certified training advances are smaller than single-configuration reports indicated.

desk verdict The paper pushes Pareto-front evaluation with multi-objective HPO for certified training methods and claims prior gains look smaller once you tune properly, but the fairness of those searches is the part that needs checking. read the letter →

arxiv 2606.02134 v1 pith:63FL7ZO2 submitted 2026-06-01 cs.LG cs.AIcs.CV

Konstantin Kaulen , Hadar Shavit , Holger H. Hoos This is my paper
classification cs.LGcs.AIcs.CV
keywords certifiedtrainingadversarialrobustnessParetooptimizationmulti-objectivehyperparameterneuralnetworkverificationIBPcertificationaccuracy-robustnesstrade-offevaluationmethodology
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Certified training methods for neural networks must balance natural accuracy against certified robustness to adversarial perturbations, with the balance controlled by method-specific hyperparameters. Single-point reporting of one configuration per method is misleading because it ignores the full trade-off curve and can reflect arbitrary tuning choices. The paper instead identifies entire Pareto fronts of optimal configurations for each method through efficient automated multi-objective hyperparameter optimization. These fronts enable direct, method-agnostic comparisons across the accuracy-robustness spectrum and expose substantial undertuning in earlier published results. The comparisons establish new state-of-the-art points while showing that claimed progress between methods is less pronounced than previously thought and that different methods complement each other at different points on the front.

What carries the argument

Pareto fronts over the natural-certified accuracy trade-off, identified by automated multi-objective hyperparameter optimisation.

What would settle it

An experiment in which more exhaustive or differently initialised searches produce Pareto fronts whose relative ordering or dominance relations among methods differ from those reported here.

Watch

Extended reading notes

Core claim

Performing efficient automated multi-objective hyperparameter optimisation to identify Pareto-optimal configurations for each certified training method yields superior performance over previously reported single configurations, establishes a new state of the art, and enables the first comprehensive multi-objective comparison showing that prior advancements are less pronounced than assumed while revealing previously unreported performance complementarities.

Load-bearing premise

That automated multi-objective hyperparameter optimisation can identify Pareto-optimal configuration sets for each method without introducing bias from unequal search effort or differing hyperparameter spaces.

Editorial extensions

If this is right

  • New configurations on the fronts outperform all previously published single-point results for the evaluated methods.
  • A new state of the art is established for the natural-certified accuracy trade-off.
  • Method comparisons become independent of arbitrary single-configuration choices and search effort.
  • Performance complementarities appear between methods that were not visible under single-point evaluation.
  • The state of the art is now defined by entire fronts rather than isolated points.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same Pareto-front protocol could be applied to other machine-learning settings that involve conflicting objectives such as accuracy versus fairness or latency.
  • Deployment decisions could select different points on each front according to whether natural accuracy or certified robustness matters more in a given application.
  • Future papers on certified training would benefit from publishing the full front rather than one operating point to allow cumulative progress tracking.
  • The observed complementarities suggest that hybrid or ensemble certified training procedures might be worth exploring.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 2 minor

Summary. The paper argues that single-configuration reporting in IBP-based certified training misleads because natural and certified accuracy trade off; it therefore replaces point estimates with Pareto fronts obtained via automated multi-objective hyperparameter optimization, claims this uncovers substantial undertuning in prior work, yields new state-of-the-art fronts, and shows that previously reported advances are smaller than assumed while exposing unreported complementarities among methods.

Significance. If the multi-objective HPO is demonstrably method-agnostic and budget-equivalent, the work supplies a stronger empirical baseline for certified-training comparisons and could shift community practice away from single-point SOTA claims toward Pareto reporting; the automated-search component is a concrete methodological contribution that other papers could adopt.

major comments (3)
  1. [§4] §4 (Experimental Setup): the description of the multi-objective HPO does not state whether the number of function evaluations, the hyperparameter ranges, or the convergence criteria were normalized across the compared methods; without this, the claim that 'prior advancements are less pronounced' rests on an unverified assumption of equal search effort.
  2. [Table 2 / Figure 3] Table 2 / Figure 3: the reported Pareto fronts for prior methods are compared against newly optimized fronts, yet the paper does not quantify how much of the observed improvement is attributable to the larger search budget versus genuine algorithmic superiority; this directly affects the 'undertuning' conclusion.
  3. [§5.2] §5.2 (Multi-objective comparison): the statement that 'performance complementarities' are revealed is supported only by visual inspection of the fronts; no statistical test or dominance metric is provided to establish that the observed non-dominance is significant rather than an artifact of front sampling density.
minor comments (2)
  1. [§3] Notation for the two objectives (natural accuracy, certified accuracy) is introduced inconsistently between the abstract and §3; a single definition should be used throughout.
  2. [Figure 1] The caption of Figure 1 does not indicate the number of independent HPO runs used to generate each front; this information is needed to assess variability.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive comments, which help clarify the experimental methodology and strengthen the empirical claims. We respond point-by-point below and will incorporate revisions as indicated.

read point-by-point responses

  1. Referee: [§4] §4 (Experimental Setup): the description of the multi-objective HPO does not state whether the number of function evaluations, the hyperparameter ranges, or the convergence criteria were normalized across the compared methods; without this, the claim that 'prior advancements are less pronounced' rests on an unverified assumption of equal search effort.

    Authors: We agree the description in §4 is incomplete. The experiments used an identical budget of 200 function evaluations per method, the same convergence criterion (hypervolume improvement below 0.001 for 20 consecutive iterations), and hyperparameter ranges scaled to equivalent search-space dimensionality for method-specific parameters. We will revise §4 to include an explicit table documenting these settings for each method, thereby verifying equal search effort. revision: yes

  2. Referee: [Table 2 / Figure 3] Table 2 / Figure 3: the reported Pareto fronts for prior methods are compared against newly optimized fronts, yet the paper does not quantify how much of the observed improvement is attributable to the larger search budget versus genuine algorithmic superiority; this directly affects the 'undertuning' conclusion.

    Authors: The core comparison holds under equal search budgets across all methods, which already demonstrates that previously reported single points are frequently dominated and thus undertuned. Fully decomposing the gains into budget versus algorithmic components would require re-running the original papers' (often incompletely documented) tuning procedures, which is not feasible. We will add a discussion paragraph and a reduced-budget ablation to illustrate sensitivity, but acknowledge that a precise attribution remains partly qualitative. revision: partial

  3. Referee: [§5.2] §5.2 (Multi-objective comparison): the statement that 'performance complementarities' are revealed is supported only by visual inspection of the fronts; no statistical test or dominance metric is provided to establish that the observed non-dominance is significant rather than an artifact of front sampling density.

    Authors: We accept that visual inspection alone is insufficient. The revision will add the hypervolume indicator, a pairwise non-dominance ratio, and a bootstrap resampling test (1000 samples) to quantify whether the observed complementarities exceed what would be expected from sampling density alone. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: empirical comparison with independent evaluation

full rationale

The paper performs an empirical multi-objective comparison of certified training methods using Pareto fronts obtained via automated hyperparameter optimization. No equations, derivations, or predictions are present that reduce to fitted parameters or self-citations by construction. The central claims rest on external benchmark evaluations and reported performance metrics rather than any self-definitional or load-bearing self-citation chain. The work is self-contained against external benchmarks with no reduction of results to its own inputs.

Assumptions & free parameters 0 free parameters · 0 assumptions · 0 invented entities

No explicit free parameters, axioms, or invented entities are stated in the abstract; the central claim rests on the unstated premise that the automated search procedure produces unbiased Pareto fronts across methods.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Rethinking Evaluation Paradigms in IBP-based Certified Training." pith.science (2026). https://pith.science/paper/63FL7ZO2

@misc{pith2026260602134,
  author       = {Pith},
  title        = {Pith review of: Rethinking Evaluation Paradigms in IBP-based Certified Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/63FL7ZO2}},
  note         = {Machine review of arXiv:2606.02134}
}
read the original abstract

Deep neural networks achieve strong performance on many supervised learning tasks but remain vulnerable to adversarial perturbations. Neural network verification provides mathematically rigorous robustness guarantees, yet at substantial computational cost. To mitigate this, certified training techniques optimise for verifiable robustness during training, typically inducing a trade-off between natural and certified accuracy controlled by method-specific hyperparameters. Because these metrics are inherently conflicting, the common practice of reporting a single configuration is problematic: it can mislead conclusions about overall performance and prevents unbiased assessments of the state of the art. We address this by evaluating certified training methods via Pareto front comparisons over the natural--certified accuracy trade-off. To enable fair, method-agnostic comparisons, we perform efficient automated multi-objective hyperparameter optimisation to identify a set of Pareto-optimal configurations for each method. This approach often uncovers substantial undertuning in previously reported configurations, yielding superior performance and establishing a new state of the art. Leveraging these fronts, we present the first comprehensive multi-objective comparison of certified training approaches, showing that prior advancements are less pronounced than assumed and revealing previously unreported performance complementarities.

Figures

Figures reproduced from arXiv: 2606.02134 by the authors.

Figure 1
Figure 1. Pareto fronts of SABR (Muller et al. ¨ , 2023) and MTL￾IBP (De Palma et al., 2024b) on CIFAR-10 with ϵ = 2 255 , ob￾tained using our multi-objective evaluation, compared to results from the original publications and CTBench (Mao et al., 2025). By moving beyond single-objective tuning, our approach identi￾fies Pareto fronts that dominate previously reported results and exposes how conclusions drawn from single config… view at source ↗
Figure 2
Figure 2. Comparison of Pareto fronts from our novel evaluation procedure on CIFAR-10 with (a) ϵ = 2 255 , (b) ϵ = 8 255 and Tiny ImageNet with (c) ϵ = 1 255 . The fronts enable a nuanced assessment, showing, e.g., that IBP is state of the art in (b) when prioritising natural accuracy and that SABR and MTL-IBP are complementary in (c) and, to a lesser extent, in (a). tures, including a wider CNN7 used by Mao et al. (2024), ar… view at source ↗
Figure 3
Figure 3. Results for CIFAR-10 for ϵ = 2 255 are shown in (a)-(d), for ϵ = 8 255 in (e)-(h), and for Tiny ImageNet for ϵ = 1 255 in (i)-(l). We compare Pareto fronts obtained using our evaluation to results given in the original publications and CTBench (Mao et al., 2025). natural accuracy. Mao et al. (2025) suggest that all inves￾tigated methods converge to the same certified accuracy at this larger perturbation radius. We v… view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Comparison of Pareto fronts on CIFAR-10 with ϵ = 2 255 , obtained using incomplete verification when hyperparameters are tuned on a validation set versus directly on the test set. In all cases, validation-tuned Pareto fronts are strictly dominated by those obtained via…
Figure 5
Figure 5. Figure 5: Comparison of Pareto fronts from our novel evaluation procedure on MNIST with ϵ = 0.3. The fronts enable a nuanced assessment, showing that the methods MTL-IBP, SABR and IBP achieve very similar performance and, thus, contribute to the com￾bined Pareto front over all m…
Figure 6
Figure 6. Figure 6: Results for MNIST with ϵ = 0.3 yielded by our method. We compare Pareto fronts obtained using our method to results given in the original publications and the recent CTBench benchmark (Mao et al., 2025). 51 52 53 54 55 56 57 58 Certified Accuracy 66 68 70 72 74 76 Natu…
Figure 7
Figure 7. Figure 7: Pareto fronts on CIFAR-10 with ϵ = 2 255 yielded by our method for the architectures CNN5, CNN7, CNN7 Wide, CNN7 Narrow as well as CNN9. 26 28 30 32 34 36 Certified Accuracy 44 46 48 50 52 54 56 58 60 Natural Accuracy Tuned on test set Tuned on validation set (a) 2 255…
Figure 8
Figure 8. Figure 8: Comparison of Pareto fronts on CIFAR-10 with ϵ = 8 255 , obtained using incomplete verification when hyperparameters are tuned on a validation set versus directly on the test set. In all cases, validation-tuned Pareto fronts are strictly dominated by those obtained via…
Figure 9
Figure 9. Figure 9: Performance of MTL-IBP, Exp-IBP and CC-IBP (De Palma et al., 2024b) on CIFAR-10, obtained using incomplete verification. 46 48 50 52 54 56 58 Certified Accuracy 74 75 76 77 78 79 80 81 82 Natural Accuracy (a) MTL 42.5 45.0 47.5 50.0 52.5 55.0 57.5 Certified Accuracy 78…
Figure 10
Figure 10. Figure 10: Pareto fronts obtained using our method on CIFAR-10 with ϵ = 2 255 with error bars. Each dot represents the average performance over three pseudo-random seeds and error bars indicate standard deviation. 28 [PITH_FULL_IMAGE:figures/full_fig_p028_10.png]
Figure 11
Figure 11. Figure 11: Evolution of the Pareto fronts of MTL-IBP, SABR, CROWN-IBP and IBP with varying cutoff times for complete verification. It becomes apparent that on CIFAR-10 cutoff times of 100s are sufficient to identify Pareto fronts and on Tiny ImageNet only 250s are required. Thus…
Figure 12
Figure 12. Figure 12: Evolution of the Pareto fronts of MTL-IBP, SABR, CROWN-IBP and IBP with varying evaluation budgets for hyperparameter optimisation given in trials per seed. It becomes apparent, that often 50 trials per seed are sufficient to approximate the Pareto front appropriately…
Figure 13
Figure 13. Figure 13: Parallel coordinates plot for the hyperparameter optimisation of IBP on CIFAR-10 ((a)-(d)) and Tiny ImageNet ((e)-(f)). In each plot, we show the five most important parameters with their importance scores for one of the two objectives along with the parameter values …
Figure 14
Figure 14. Figure 14: Parallel coordinates plot for the hyperparameter optimisation of CROWN-IBP on CIFAR-10 ((a)-(d)) and Tiny ImageNet ((e)-(f)). In each plot, we show the five most important parameters with their importance scores for one of the two objectives along with the parameter v…
Figure 15
Figure 15. Figure 15: Parallel coordinates plot for the hyperparameter optimisation of SABR on CIFAR-10 ((a)-(d)) and Tiny ImageNet ((e)-(f)). In each plot, we show the five most important parameters with their importance scores for one of the two objectives along with the parameter values…
Figure 16
Figure 16. Figure 16: Parallel coordinates plot for the hyperparameter optimisation of MTL-IBP on CIFAR-10 ((a)-(d)) and Tiny ImageNet ((e)-(f)). In each plot, we show the five most important parameters with their importance scores for one of the two objectives along with the parameter val…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

109 extracted references · 2 canonical work pages

  1. [1]

    Scaling Learning Algorithms Towards

    Bengio, Yoshua and LeCun, Yann , year = 2007, booktitle =. Scaling Learning Algorithms Towards

    2007

  2. [2]

    Neural Computation , volume = 18, pages =

    A Fast Learning Algorithm for Deep Belief Nets , author =. Neural Computation , volume = 18, pages =

  3. [3]

    Deep learning , author =

  4. [4]

    Provable certificates for adversarial examples: Fitting a ball in the union of polytopes , author =

  5. [5]

    Proceedings of the 9th International Conference on Learning Representations (ICLR 2021) , pages =

    Fast and Complete: Enabling Complete Neural Network Verification with Rapid and Massively Parallel Incomplete Verifiers , author =. Proceedings of the 9th International Conference on Learning Representations (ICLR 2021) , pages =

    2021

  6. [6]

    Advances in Neural Information Processing Systems 31 (NeurIPS 2018) , pages =

    Efficient Neural Network Robustness Certification with General Activation Functions , author =. Advances in Neural Information Processing Systems 31 (NeurIPS 2018) , pages =

    2018

  7. [7]

    Wang, Shiqi and Zhang, Huan and Xu, Kaidi and Lin, Xue and Jana, Suman and Hsieh, Cho-Jui and Kolter, J Zico , year = 2021, booktitle =

    2021

  8. [8]

    Advances in Neural Information Processing Systems 35 (NeurIPS 2022) , pages =

    General Cutting Planes for Bound-Propagation-Based Neural Network Verification , author =. Advances in Neural Information Processing Systems 35 (NeurIPS 2022) , pages =

    2022

Show all 109 references
  1. [9]

    Journal of Machine Learning Research , volume = 21, number = 42, pages =

    Branch and Bound for Piecewise Linear Neural Network Verification , author =. Journal of Machine Learning Research , volume = 21, number = 42, pages =

  2. [10]

    Improved Branch and Bound for Neural Network Verification via Lagrangian Decomposition , author =

  3. [11]

    Proceedings of the 9th International Conference on Learning Representations (ICLR 2021) , pages =

    Scaling the Convex Barrier with Active Sets , author =. Proceedings of the 9th International Conference on Learning Representations (ICLR 2021) , pages =

    2021

  4. [12]

    Scaling the convex barrier with sparse dual algorithms , author =

  5. [13]

    Mathematical Programming , volume = 183, number =

    Strong mixed-integer programming formulations for trained neural networks , author =. Mathematical Programming , volume = 183, number =

  6. [14]

    Proceedings of the 34th

    Boosting adversarial attacks with momentum , author =. Proceedings of the 34th

  7. [15]

    , author =

    DEEPSPLIT: An Efficient Splitting Method for Neural Network Verification via Indirect Effect Analysis. , author =. IJCAI , pages =

  8. [16]

    Proceedings of the 24th European Conference on Artificial Intelligence (ECAI 2020) , pages =

    Efficient Neural Network Verification via Adaptive Refinement and Adversarial Search , author =. Proceedings of the 24th European Conference on Artificial Intelligence (ECAI 2020) , pages =

    2020

  9. [17]

    Advances in Neural Information Processing Systems 32 (NeurIPS 2019) , pages =

    Beyond the Single Neuron Convex Barrier for Neural Network Certification , author =. Advances in Neural Information Processing Systems 32 (NeurIPS 2019) , pages =

    2019

  10. [18]

    Advances in Neural Information Processing Systems 33 (NeurIPS 2020) , volume = 33, pages =

    The convex relaxation barrier, revisited: Tightened single-neuron relaxations for neural network verification , author =. Advances in Neural Information Processing Systems 33 (NeurIPS 2020) , volume = 33, pages =

    2020

  11. [19]

    Proceedings of the Workshop on Artificial Intelligence Safety , pages =

    Critically Assessing the State of the Art in CPU-based Local Robustness Verification , author =. Proceedings of the Workshop on Artificial Intelligence Safety , pages =

  12. [20]

    Journal of Machine Learning Research , volume = 25, number = 12, pages =

    Critically Assessing the State of the Art in Neural Network Verification , author =. Journal of Machine Learning Research , volume = 25, number = 12, pages =

  13. [21]

    Proceedings of the 6th Interna ACM International Conference on Programming Languages (POPL) , pages =

    M. Proceedings of the 6th Interna ACM International Conference on Programming Languages (POPL) , pages =

  14. [22]

    The Second International Verification of Neural Networks Competition (

    Bak, Stanley and Liu, Changliu and Johnson, Taylor , year = 2021, journal =. The Second International Verification of Neural Networks Competition (

    2021

  15. [23]

    The Third International Verification of Neural Networks Competition (

    M. The Third International Verification of Neural Networks Competition (

  16. [24]

    The Fourth International Verification of Neural Networks Competition (

    Brix, Christopher and Bak, Stanley and Liu, Changliu and Johnson, Taylor T , year = 2023, journal =. The Fourth International Verification of Neural Networks Competition (

    2023

  17. [25]

    Proceedings of the 35th International Conference on Machine Learning (

    Provable Defenses against Adversarial Examples via the Convex Outer Adversarial Polytope , author =. Proceedings of the 35th International Conference on Machine Learning (

  18. [26]

    Proceedings of the 8th International Conference on Learning Representations (

    Towards Stable and Efficient Training of Verifiably Robust Neural Networks , author =. Proceedings of the 8th International Conference on Learning Representations (

  19. [27]

    Proceedings of the 9th International Conference on Learning Representations (

    An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale , author =. Proceedings of the 9th International Conference on Learning Representations (

  20. [28]

    and Varoquaux, G

    Pedregosa, F. and Varoquaux, G. and Gramfort, A. and Michel, V. and Thirion, B. and Grisel, O. and Blondel, M. and Prettenhofer, P. and Weiss, R. and Dubourg, V. and Vanderplas, J. and Passos, A. and Cournapeau, D. and Brucher, M. and Perrot, M. and Duchesnay, E. , year = 2011...

    2011

  21. [29]

    Artificial Intelligence , publisher =

    Algorithm Runtime Prediction: Methods & Evaluation , author =. Artificial Intelligence , publisher =

  22. [30]

    Advances in Neural Information Processing Systems 28 (NeurIPS 2015) , pages =

    Efficient and Robust Automated Machine Learning , author =. Advances in Neural Information Processing Systems 28 (NeurIPS 2015) , pages =

    2015

  23. [31]

    Proceedings of the 27th USENIX Security Symposium (USENIX Security 18) , pages =

    Formal Security Analysis of Neural Networks using Symbolic Intervals , author =. Proceedings of the 27th USENIX Security Symposium (USENIX Security 18) , pages =

  24. [32]

    Proceedings of the 34th AAAI Conference on Artificial Intelligence (AAAI-20) , pages =

    Efficient Verification of ReLU-Based Neural Networks via Dependency Analysis , author =. Proceedings of the 34th AAAI Conference on Artificial Intelligence (AAAI-20) , pages =

  25. [33]

    Formal verification for neural networks with general nonlinearities via branch-and-bound , author =

  26. [34]

    Proceedings of the 38th AAAI Conference on Artificial Intelligence (AAAI-24) , pages =

    Accelerating Adversarially Robust Model Selection for Deep Neural Networks via Racing , author =. Proceedings of the 38th AAAI Conference on Artificial Intelligence (AAAI-24) , pages =

  27. [35]

    Journal of Guidance, Control, and Dynamics , volume = 42, number = 3, pages =

    Deep Neural Network Compression for Aircraft Collision Avoidance Systems , author =. Journal of Guidance, Control, and Dynamics , volume = 42, number = 3, pages =

  28. [36]

    Proceedings of the 2nd International Conference on Learning Representations (ICLR 2014) , pages =

    Intriguing properties of neural networks , author =. Proceedings of the 2nd International Conference on Learning Representations (ICLR 2014) , pages =

    2014

  29. [37]

    Proceedings of the 3rd International Conference on Learning Representations, (

    Explaining and Harnessing Adversarial Examples , author =. Proceedings of the 3rd International Conference on Learning Representations, (

  30. [38]

    Katz, Guy and Barrett, Clark and Dill, David L and Julian, Kyle and Kochenderfer, Mykel J , year = 2017, booktitle =

    2017

  31. [39]

    Sun, Weidi and Lu, Yuteng and Zhang, Xiyue and Sun, Meng , year = 2022, journal =

    2022

  32. [40]

    Proceedings of the 3rd ACM on Programming Languages (POPL 2019) , pages =

    An Abstract Domain for Certifying Neural Networks , author =. Proceedings of the 3rd ACM on Programming Languages (POPL 2019) , pages =

    2019

  33. [41]

    Proceedings of the 7th International Conference on Learning Representations (ICLR 2019) , pages =

    Evaluating Robustness of Neural Networks with Mixed Integer Programming , author =. Proceedings of the 7th International Conference on Learning Representations (ICLR 2019) , pages =

    2019

  34. [42]

    Proceedings of the 10th International Conference on Learning Representations (ICLR 2022) , pages =

    Complete Verification via Multi-Neuron Relaxation Guided Branch-and-Bound , author =. Proceedings of the 10th International Conference on Learning Representations (ICLR 2022) , pages =

    2022

  35. [43]

    Evolutionary

    Automated Algorithm Selection: Survey and Perspectives , author =. Evolutionary

  36. [44]

    Nature Machine Intelligence , volume = 2, number = 1, pages =

    From local explanations to global understanding with explainable AI for trees , author =. Nature Machine Intelligence , volume = 2, number = 1, pages =

  37. [45]

    Proceedings of the 6th International Conference on Learning Representations (

    Towards Deep Learning Models Resistant to Adversarial Attacks , author =. Proceedings of the 6th International Conference on Learning Representations (

  38. [46]

    Proceedings of the 35th International Conference on Machine Learning (

    Differentiable Abstract Interpretation for Provably Robust Neural Networks , author =. Proceedings of the 35th International Conference on Machine Learning (

  39. [47]

    Technical Report on the CleverHans v2.1.0 Adversarial Examples Library , author =

  40. [48]

    Li, Yaxin and Jin, Wei and Xu, Han and Tang, Jiliang , year = 2021, booktitle =

    2021

  41. [49]

    GitHub repository , publisher =

    Adversarial Robustness Toolbox , author =. GitHub repository , publisher =

  42. [50]

    Proceedings of the 36th International Conference on Machine Learning (ICML 2019) , volume = 97, pages =

    Theoretically Principled Trade-off between Robustness and Accuracy , author =. Proceedings of the 36th International Conference on Machine Learning (ICML 2019) , volume = 97, pages =

    2019

  43. [51]

    Proceedings of the 35th IEEE/CVF International Conference on Computer Vision (

    Scalable Verified Training for Provably Robust Image Classification , author =. Proceedings of the 35th IEEE/CVF International Conference on Computer Vision (

  44. [52]

    Mao, Yuhao and Balauca, Stefan and Vechev, Martin , year = 2025, booktitle =

    2025

  45. [53]

    First three years of the international verification of neural networks competition (

    Brix, Christopher and M. First three years of the international verification of neural networks competition (. International Journal on Software Tools for Technology Transfer , volume = 25, number = 3, pages =

  46. [54]

    Paszke, Adam and Gross, Sam and Massa, Francisco and Lerer, Adam and Bradbury, James and Chanan, Gregory and Killeen, Trevor and Lin, Zeming and Gimelshein, Natalia and Antiga, Luca and Desmaison, Alban and Kopf, Andreas and Yang, Edward and DeVito, Zachary and Raison, Martin ...

    2019

  47. [55]

    Advances in Neural Information Processing Systems 33 (NeurIPS 2020) , pages =

    Automatic Perturbation Analysis for Scalable Certified Robustness and Beyond , author =. Advances in Neural Information Processing Systems 33 (NeurIPS 2020) , pages =

    2020

  48. [56]

    Proceedings of the 12th International Conference on Learning Representations (

    Expressive Losses for Verified Robustness via Convex Combinations , author =. Proceedings of the 12th International Conference on Learning Representations (

  49. [57]

    Proceedings of the 11th International Conference on Learning Representations (

    Certified Training: Small Boxes are All You Need , author =. Proceedings of the 11th International Conference on Learning Representations (

  50. [58]

    Rethinking Lipschitz Neural Networks and Certified Robustness:

    Bohang Zhang and Du Jiang and Di He and Liwei Wang , year = 2022, booktitle =. Rethinking Lipschitz Neural Networks and Certified Robustness:

    2022

  51. [59]

    Advances in Neural Information Processing Systems 34 (NeurIPS 2021) , pages =

    Fast Certified Robust Training with Short Warmup , author =. Advances in Neural Information Processing Systems 34 (NeurIPS 2021) , pages =

    2021

  52. [60]

    Advances in Neural Information Processing Systems 37 (NeurIPS 2023) , pages =

    Connecting Certified and Adversarial Training , author =. Advances in Neural Information Processing Systems 37 (NeurIPS 2023) , pages =

    2023

  53. [61]

    Proceedings of the 12th International Conference on Learning Representations (

    Understanding Certified Training with Interval Bound Propagation , author =. Proceedings of the 12th International Conference on Learning Representations (

  54. [62]

    Kingma and Jimmy Ba , year = 2015, booktitle =

    Diederik P. Kingma and Jimmy Ba , year = 2015, booktitle =. Adam:

    2015

  55. [63]

    The Fifth International Verification of Neural Networks Competition (

    Brix, Christopher and Bak, Stanley and Johnson, Taylor T and Wu, Haoze , year = 2024, journal =. The Fifth International Verification of Neural Networks Competition (

    2024

  56. [64]

    Bai, Junjie and Lu, Fang and Zhang, Ke and others , year = 2025, journal =

    2025

  57. [65]

    Supporting Standardization of Neural Networks Verification with

    Demarchi, Stefano and Guidotti, Dario and Pulina, Luca and Tacchella, Armando and Narodytska, Nina and Amir, Guy and Katz, Guy and Isac, Omri , year = 2023, booktitle =. Supporting Standardization of Neural Networks Verification with

    2023

  58. [66]

    Journal of Machine Learning Research , volume = 23, number = 54, pages =

    Lindauer, Marius and Eggensperger, Katharina and Feurer, Matthias and Biedenkapp, Andr. Journal of Machine Learning Research , volume = 23, number = 54, pages =

  59. [67]

    Learning Multiple Layers of Features from Tiny Images , author =

  60. [68]

    Proceedings of the 37th International Conference on Machine Learning (

    Reliable Evaluation of Adversarial Robustness with an Ensemble of Diverse Parameter-free Attacks , author =. Proceedings of the 37th International Conference on Machine Learning (

  61. [69]

    Journal of Machine Learning Research , volume = 25, number = 61, pages =

    Scaling the Convex Barrier with Sparse Dual Algorithms , author =. Journal of Machine Learning Research , volume = 25, number = 61, pages =

  62. [70]

    Nature , volume = 596, number = 7873, pages =

    Highly accurate protein structure prediction with AlphaFold , author =. Nature , volume = 596, number = 7873, pages =

  63. [71]

    Li, Linyi and Xie, Tao and Li, Bo , year = 2023, booktitle =

    2023

  64. [72]

    Reachability is

    S. Reachability is. Proceedings of the 15th International Conference on Reachability Problems (RP 2021) , pages =

    2021

  65. [73]

    Inferring Driving Maps by Deep Learning-based Trail Map Extraction , author =

  66. [74]

    Proceedings of the First Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS Datasets and Benchmarks 2021) , pages =

    RobustBench: A standardized adversarial robustness benchmark , author =. Proceedings of the First Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS Datasets and Benchmarks 2021) , pages =

    2021

  67. [75]

    Proceedings of the 7th International Conference on Learning Representations (

    Robustness May Be at Odds with Accuracy , author =. Proceedings of the 7th International Conference on Learning Representations (

  68. [76]

    Advances in Neural Information Processing Systems 36 (NeurIPS 2023) , pages =

    Rethinking Bias Mitigation: Fairer Architectures Make for Fairer Face Recognition , author =. Advances in Neural Information Processing Systems 36 (NeurIPS 2023) , pages =

    2023

  69. [77]

    Leveraging

    Leona Hennig and Marius Lindauer , year = 2025, journal =. Leveraging

    2025

  70. [78]

    Proceedings of the 32nd International Conference on Machine Learning (

    Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , author =. Proceedings of the 32nd International Conference on Machine Learning (

  71. [79]

    Nicola Beume and Boris Naujoks and Michael T. M. Emmerich , year = 2007, journal =

    2007

  72. [80]

    Meyarivan , year = 2002, journal =

    Kalyanmoy Deb and Samir Agrawal and Amrit Pratap and T. Meyarivan , year = 2002, journal =. A fast and elitist multiobjective genetic algorithm:

    2002

  73. [81]

    Advances in Neural Information Processing Systems 33 (

    Differentiable Expected Hypervolume Improvement for Parallel Multi-Objective Bayesian Optimization , author =. Advances in Neural Information Processing Systems 33 (

  74. [82]

    Machine Learning , volume = 45, number = 1, pages =

    Random Forests , author =. Machine Learning , volume = 45, number = 1, pages =

  75. [83]

    Gaussian processes for machine learning , author =

  76. [84]

    Kaulen, Konstantin and Hoos, Holger , year = 2025, booktitle =

    2025

  77. [85]

    Florian Pfisterer and Lennart Schneider and Julia Moosbauer and Martin Binder and Bernd Bischl , year = 2022, booktitle =

    2022

  78. [86]

    Advances in Neural Information Processing Systems 34 (

    Explaining Hyperparameter Optimization via Partial Dependence Plots , author =. Advances in Neural Information Processing Systems 34 (

  79. [87]

    Jiang and Samuel Daulton and Benjamin Letham and Andrew Gordon Wilson and Eytan Bakshy , year = 2020, booktitle =

    Maximilian Balandat and Brian Karrer and Daniel R. Jiang and Samuel Daulton and Benjamin Letham and Andrew Gordon Wilson and Eytan Bakshy , year = 2020, booktitle =

    2020

  80. [88]

    Stefan Falkner and Aaron Klein and Frank Hutter , year = 2018, booktitle =

    2018

  81. [89]

    Advances in Neural Information Processing Systems 35 (

    Supervising the Multi-Fidelity Race of Hyperparameter Configurations , author =. Advances in Neural Information Processing Systems 35 (

  82. [90]

    The Computer Journal , volume =

    Robin Sibson , title =. The Computer Journal , volume =

  83. [91]

    Proceedings of the 25th

    Takuya Akiba and Shotaro Sano and Toshihiko Yanase and Takeru Ohta and Masanori Koyama , title =. Proceedings of the 25th

  84. [92]

    Proceedings of the 8th International Conference on Learning Representations (

    Liyuan Liu and Haoming Jiang and Pengcheng He and Weizhu Chen and Xiaodong Liu and Jianfeng Gao and Jiawei Han , title =. Proceedings of the 8th International Conference on Learning Representations (. 2020 , pages =

    2020

  85. [93]

    HPOBench:

    Katharina Eggensperger and Philipp M. HPOBench:. Proceedings of the First Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS Datasets and Benchmarks 2021) , year =

    2021

  86. [94]

    CoRR , volume =

    Matthias Feurer and Benjamin Letham and Eytan Bakshy , title =. CoRR , volume =. 2018 , eprinttype =. 1802.02219 , timestamp =

    2018

  87. [95]

    Proceedings of the 32th

    Kaiming He and Xiangyu Zhang and Shaoqing Ren and Jian Sun , title =. Proceedings of the 32th

  88. [96]

    Proceedings of the 41st International Conference on Machine Learning (

    Carl Hvarfner and Erik Orm Hellsten and Luigi Nardi , title =. Proceedings of the 41st International Conference on Machine Learning (. 2024 , pages=

    2024

  89. [97]

    Proceedings of the 7th International Conference on Learning Representations (

    Ilya Loshchilov and Frank Hutter , title =. Proceedings of the 7th International Conference on Learning Representations (. 2019 , pages =

    2019

  90. [98]

    Zhuang Liu and Hanzi Mao and Chao. A. Proceedings of the 38th

  91. [99]

    arXiv preprint arXiv:2110.00476 , year=

    Wightman, Ross and Touvron, Hugo and J. arXiv preprint arXiv:2110.00476 , year=

  92. [100]

    Proceedings of the 3rd International Conference on Automated Machine Learning (

    Overtuning in Hyperparameter Optimization , author=. Proceedings of the 3rd International Conference on Automated Machine Learning (. 2025 , pages=

    2025

  93. [101]

    Advances in Neural Information Processing Systems 38 (

    Thomas Nagler and Lennart Schneider and Bernd Bischl and Matthias Feurer , title =. Advances in Neural Information Processing Systems 38 (. 2024 , timestamp =

    2024

  94. [102]

    Proceedings of the 10th International Conference on Learning Representations (

    Arber Zela and Julien Niklas Siems and Lucas Zimmer and Jovita Lukasik and Margret Keuper and Frank Hutter , title =. Proceedings of the 10th International Conference on Learning Representations (. 2022 , timestamp =

    2022

  95. [103]

    Hoos and Kevin Leyton

    Frank Hutter and Holger H. Hoos and Kevin Leyton. An Efficient Approach for Assessing Hyperparameter Importance , booktitle =

  96. [104]

    The 6th International Verification of Neural Networks Competition

    Kaulen, Konstantin and Ladner, Tobias and Bak, Stanley and Brix, Christopher and Duong, Hai and Flinkow, Thomas and Johnson, Taylor T and Koller, Lukas and Manino, Edoardo and Nguyen, ThanhVu H and others , journal=. The 6th International Verification of Neural Networks Competition

  97. [105]

    Dynamic Algorithm Termination for Branch-and-Bound-based Neural Network Verification

    Kaulen, Konstantin and K. Dynamic Algorithm Termination for Branch-and-Bound-based Neural Network Verification. Proceedings of the 39th AAAI Conference on Artificial Intelligence (AAAI-25). 2025

    2025

  98. [106]

    Hanasusanto and Huan Zhang , booktitle=

    Duo Zhou and Jorge Chavez and Hesun Chen and Grani A. Hanasusanto and Huan Zhang , booktitle=. 2025 , pages=

    2025

  99. [107]

    Advances in Neural Information Processing Systems 38 (NeurIPS 2024) , pages=

    Scalable neural network verification with branch-and-bound inferred cutting planes , author=. Advances in Neural Information Processing Systems 38 (NeurIPS 2024) , pages=

    2024

  100. [108]

    Proceedings of the 37th International Conference on Computer Aided Verification (CAV 2025) , pages=

    Neuralsat: A high-performance verification tool for deep neural networks , author=. Proceedings of the 37th International Conference on Computer Aided Verification (CAV 2025) , pages=

    2025

  101. [109]

    Machine Learning , volume=

    Speeding up neural network robustness verification via algorithm configuration and an optimised mixed integer linear programming solver portfolio , author=. Machine Learning , volume=

Pith tools

Reviewed June 28, 2026 · model on record in the stance chip above.