A new no-free-lunch construction and empirical scaling study argue that robust classification can require exponentially more data than standard classification, and that dataset size largely drives certified robust accuracy.
1-Lipschitz Neural Networks are more expressive with N-Activations
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
A crucial property for achieving secure, trustworthy and interpretable deep learning systems is their robustness: small changes to a system's inputs should not result in large changes to its outputs. Mathematically, this means one strives for networks with a small Lipschitz constant. Several recent works have focused on how to construct such Lipschitz networks, typically by imposing constraints on the weight matrices. In this work, we study an orthogonal aspect, namely the role of the activation function. We show that commonly used activation functions, such as MaxMin, as well as all piece-wise linear ones with two segments unnecessarily restrict the class of representable functions, even in the simplest one-dimensional setting. We furthermore introduce the new N-activation function that is provably more expressive than currently popular activation functions. We provide code at https://github.com/berndprach/NActivation.
citation-role summary
background 1
citation-polarity summary
fields
cs.CV 1years
2024 1verdicts
CONDITIONAL 1roles
background 1polarities
unclear 1representative citing papers
citing papers explorer
-
Intriguing Properties of Robust Classification
A new no-free-lunch construction and empirical scaling study argue that robust classification can require exponentially more data than standard classification, and that dataset size largely drives certified robust accuracy.