RECORD
Shift Invariance Can Reduce Adversarial Robustness
Machine Learning · Submitted 3 Mar 2021
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim This is more than a curiosity; we show empirically that with real datasets and realistic architectures, shift invariance reduces adversarial robustness.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
Shift invariance is a critical property of CNNs that improves performance on classification. However, we show that invariance to circular shifts can also lead to greater sensitivity to adversarial attacks. We first characterize the margin between classes when a shift-invariant linear classifier is used. We show that the margin can only depend on the DC component of the signals. Then, using results about infinitely wide networks, we show that in some simple cases, fully connected and shift-invariant neural networks produce linear decision boundaries. Using this, we prove that shift invariance in neural networks produces adversarial examples for the simple case of two classes, each consisting of a single image with a black or white dot on a gray background. This is more than a curiosity; we show empirically that with real datasets and realistic architectures, shift invariance reduces adversarial robustness. Finally, we describe initial experiments using synthetic data to probe the source of this connection.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | Shift invariance is a critical property of CNNs that improves performance on classification. | background 1.00 | 0.06 | 0.02 | asserted 1.00 |
| s1 | However, we show that invariance to circular shifts can also lead to greater sensitivity to adversarial attacks. | result 0.88 | 0.93 | 0.89 | asserted 0.99 |
| s2 | We first characterize the margin between classes when a shift-invariant linear classifier is used. | contribution 0.54 | 0.91 | 0.03 | asserted 0.94 |
| s3 | We show that the margin can only depend on the DC component of the signals. | result 1.00 | 0.89 | 0.04 | asserted 0.98 |
| s4 | Then, using results about infinitely wide networks, we show that in some simple cases, fully connected and shift-invariant neural networks produce linear decision boundaries. | result 0.56 | 0.92 | 0.03 | asserted 0.97 |
| s5 | Using this, we prove that shift invariance in neural networks produces adversarial examples for the simple case of two classes, each consisting of a single image with a black or white dot on a gray background. | result 0.97 | 0.96 | 0.20 | asserted 1.00 |
| s6 | This is more than a curiosity; we show empirically that with real datasets and realistic architectures, shift invariance reduces adversarial robustness. | result 0.97 | 0.96 | 0.82 | asserted 1.00 |
| s7 | Finally, we describe initial experiments using synthetic data to probe the source of this connection. | contribution 0.53 | 0.94 | 0.02 | asserted 0.65 |
What was examined, and what was not
- arXiv listing abstract v3
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
Shift invariance is a critical property of CNNs that improves performance on classification. However, we show that invariance to circular shifts can also lead to greater sensitivity to adversarial attacks. We first characterize the margin between classes when a shift-invariant linear classifier is used. We show that the margin can only depend on the DC component of the signals. Then, using results about infinitely wide networks, we show that in some simple cases, fully connected and shift-invariant neural networks produce linear decision boundaries. Using this, we prove that shift invariance in neural networks produces adversarial examples for the simple case of two classes, each consisting of a single image with a black or white dot on a gray background. This is more than a curiosity; we show empirically that with real datasets and realistic architectures, shift invariance reduces adversarial robustness. Finally, we describe initial experiments using synthetic data to probe the source of this connection.
Human review
How the literature uses it
3 arXiv papers cite this in 3 sentences
Papers citing it · 1 sentence without a confident label
Cites it as background 2 papers
-
Despite extensive research, the reasons for the abundance of adversarial examples in trained neural networks are still not well understood [ref] .
Adversarial Examples Exist in Two-Layer ReLU Networks for Low Dimensional Linear Subspaces · Related Works · 0.93
-
Despite a great deal of research, it is still unclear why neural networks are so susceptible to adversarial examples [ref] .
Gradient Methods Provably Converge to Non-Robust Networks · Introduction · 0.88
What it cites · 8 sentences without a confident label
Uses the method or tool of 1 paper
-
For AlexNet and five variants of ResNet, we train the models for 200 epochs using SGD with a cosine annealing schedule [ref] .
SGDR: Stochastic Gradient Descent with Warm Restarts · More realistic networks · 1.00
Uses data or values from 1 paper
-
Datasets We consider two datasets in which FC networks are able to attain reasonable performance compared to CNNs, MNIST [ref] and Fashion-MNIST [ref] .
Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms · Shift-invariant CNNs vs. FC Networks · 0.92
Cites as background 7 papers
-
This is widely seen in the earlier network architectures such as LeNet [ref] and AlexNet [ref] while more recent ones have universally adopted a global pooling layer (e.g. ResNet [ref] , DenseNet [ref] ) to reduce the number of parameters.
One weird trick for parallelizing convolutional neural networks · Realistic architectures and shift invariance · 0.89
-
We draw on recent work that characterizes infinitely wide neural networks as kernel methods, describing FC networks with the neural tangent kernel (NTK) [ref] and CNNs with a convolutional neural tangent kernel (CNTK) [ref] .
Enhanced Convolutional Neural Tangent Kernels · Introduction · 0.82
-
In prior work it has been suggested that larger input dimension of the data [ref] or larger intrinsic dimension of the data [ref] can reduce robustness.
Adversarial Spheres · Dimension and margin · 0.84
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.