RECORD
When Not to Classify: Anomaly Detection of Attacks (ADA) on DNN Classifiers at Test Time
Machine Learning · Submitted 18 Dec 2017
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim Tested on MNIST and CIFAR-10 image databases under three prominent attack strategies, our approach outperforms previous detection methods, achieving strong ROC AUC detection accuracy on two attacks and better accuracy than recently reported for a variety of methods on the strongest (CW) attack.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
A significant threat to the recent, wide deployment of machine learning-based systems, including deep neural networks (DNNs), is adversarial learning attacks. We analyze possible test-time evasion-attack mechanisms and show that, in some important cases, when the image has been attacked, correctly classifying it has no utility: i) when the image to be attacked is (even arbitrarily) selected from the attacker's cache; ii) when the sole recipient of the classifier's decision is the attacker. Moreover, in some application domains and scenarios it is highly actionable to detect the attack irrespective of correctly classifying in the face of it (with classification still performed if no attack is detected). We hypothesize that, even if human-imperceptible, adversarial perturbations are machine-detectable. We propose a purely unsupervised anomaly detector (AD) that, unlike previous works: i) models the joint density of a deep layer using highly suitable null hypothesis density models (matched in particular to the non- negative support for RELU layers); ii) exploits multiple DNN layers; iii) leverages a "source" and "destination" class concept, source class uncertainty, the class confusion matrix, and DNN weight information in constructing a novel decision statistic grounded in the Kullback-Leibler divergence. Tested on MNIST and CIFAR-10 image databases under three prominent attack strategies, our approach outperforms previous detection methods, achieving strong ROC AUC detection accuracy on two attacks and better accuracy than recently reported for a variety of methods on the strongest (CW) attack. We also evaluate a fully white box attack on our system. Finally, we evaluate other important performance measures, such as classification accuracy, versus detection rate and attack strength.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | A significant threat to the recent, wide deployment of machine learning-based systems, including deep neural networks (DNNs), is adversarial learning attacks. | background 1.00 | 0.04 | 0.03 | asserted 1.00 |
| s1 | We analyze possible test-time evasion-attack mechanisms and show that, in some important cases, when the image has been attacked, correctly classifying it has no utility: i) when the image to be attacked is (even arbitrarily) selected from the attacker's cache; ii) when the sole recipient of the classifier's decision is the attacker. | contribution 0.50 | 0.92 | 0.42 | asserted 0.99 |
| s2 | Moreover, in some application domains and scenarios it is highly actionable to detect the attack irrespective of correctly classifying in the face of it (with classification still performed if no attack is detected). | interpretation 0.45 | 0.17 | 0.28 | asserted 0.69 |
| s3 | We hypothesize that, even if human-imperceptible, adversarial perturbations are machine-detectable. | contribution 0.37 | 0.21 | 0.09 | speculative 0.76 |
| s4 | We propose a purely unsupervised anomaly detector (AD) that, unlike previous works: i) models the joint density of a deep layer using highly suitable null hypothesis density models (matched in particular to the non- negative support for RELU layers); ii) exploits multiple DNN layers; iii) leverages a "source" and "destination" class concept, source class uncertainty, the class confusion matrix, and DNN weight information in constructing a novel decision statistic grounded in the Kullback-Leibler divergence. | method 0.74 | 0.93 | 0.11 | asserted 0.99 |
| s5 | Tested on MNIST and CIFAR-10 image databases under three prominent attack strategies, our approach outperforms previous detection methods, achieving strong ROC AUC detection accuracy on two attacks and better accuracy than recently reported for a variety of methods on the strongest (CW) attack. | result 1.00 | 0.97 | 0.53 | asserted 1.00 |
| s6 | We also evaluate a fully white box attack on our system. | method 0.55 | 0.94 | 0.03 | asserted 0.90 |
| s7 | Finally, we evaluate other important performance measures, such as classification accuracy, versus detection rate and attack strength. | contribution 0.42 | 0.93 | 0.05 | asserted 0.83 |
What was examined, and what was not
- arXiv listing abstract v2
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
A significant threat to the recent, wide deployment of machine learning-based systems, including deep neural networks (DNNs), is adversarial learning attacks. We analyze possible test-time evasion-attack mechanisms and show that, in some important cases, when the image has been attacked, correctly classifying it has no utility: i) when the image to be attacked is (even arbitrarily) selected from the attacker's cache; ii) when the sole recipient of the classifier's decision is the attacker. Moreover, in some application domains and scenarios it is highly actionable to detect the attack irrespective of correctly classifying in the face of it (with classification still performed if no attack is detected). We hypothesize that, even if human-imperceptible, adversarial perturbations are machine-detectable. We propose a purely unsupervised anomaly detector (AD) that, unlike previous works: i) models the joint density of a deep layer using highly suitable null hypothesis density models (matched in particular to the non- negative support for RELU layers); ii) exploits multiple DNN layers; iii) leverages a "source" and "destination" class concept, source class uncertainty, the class confusion matrix, and DNN weight information in constructing a novel decision statistic grounded in the Kullback-Leibler divergence. Tested on MNIST and CIFAR-10 image databases under three prominent attack strategies, our approach outperforms previous detection methods, achieving strong ROC AUC detection accuracy on two attacks and better accuracy than recently reported for a variety of methods on the strongest (CW) attack. We also evaluate a fully white box attack on our system. Finally, we evaluate other important performance measures, such as classification accuracy, versus detection rate and attack strength.
Human review
References
How the literature uses it
3 arXiv papers cite this in 28 sentences
Papers citing it · 20 sentences without a confident label
Uses its method or tool 1 paper
-
We implemented the unsupervised AD from [ref] (referred to as L-AWA-ADA) and the blurring “correction” approach from [ref] .
Adversarial Learning in Statistical Classification: A Comprehensive Review of Defenses Against Attacks · Test-Time Evasion Experiments · 1.00
Compares with it 1 paper
-
The experiments of Section [ref] will show that a purely unsupervised AD [ref] outperforms [ref] even though we (optimistically) allow [ref] 's detector to use more than one hundred labeled examples of the attack to be detected.
Adversarial Learning in Statistical Classification: A Comprehensive Review of Defenses Against Attacks · Unsupervised AD of TTEs based on an Explicit Null Hypothesis · 0.98
Cites it as support 1 paper
-
Indeed, this is what is shown in [ref] .
Adversarial Learning in Statistical Classification: A Comprehensive Review of Defenses Against Attacks · Small Perturbations for TTE Attacks: Fallacy or Requirement? · 1.00
Cites it as background 2 papers
-
Test-time evasion attacks (TTEAs) [ref]
When Not to Classify: Detection of Reverse Engineering Attacks on DNN Image Classifiers · Introduction · 1.00
-
0em * Input-space detection - Methods that seek to identify adversarial examples through an examination/manipulation of the inputs to DNNs [ref] . * Feature-space detection - Methods which typically train new detector models on intermediate or final DNN layer outputs [ref] . * Robustness Certificates - Theoretical proofs for DNN robustness under specific threat models (assumptions for both attacker and defender) [ref] . * Statistical Detection - Methods that focus on generating robust test statistics that separate clean from adversarial examples [ref] . * Network Coverage - Methods that examine neuron coverage as a means to test DNN behavior and potentially establish differences between clean and adversarial examples [ref] .
Attack Agnostic Detection of Adversarial Examples via Random Subspace Analysis · Related Work · 1.00
What it cites · 17 sentences without a confident label
Contests 2 papers
-
However, without even considering these choices, we propose an improved detector, which fundamentally exploits more information about a possible anomaly, to strong effect (as demonstrated by our results), compared to the procedure from [ref] .
Detecting Adversarial Samples from Artifacts · The Method from CITEMARK143Z · contest 0.76
-
However, [ref] failed to detect the CW attack on the CIFAR-10 image domain [ref] .
On the (Statistical) Detection of Adversarial Examples · Recent Works on Detection of Test-Time Evasion Attacks · contest 0.89
Uses the method or tool of 1 paper
-
The AD proposed in [ref] consists of the following operations, given a test pattern (image), $x$: * Determine the maximum a posteriori class under the DNN model: $c^∗ = max_c ∈{1,…,K} P[C=c | x]$. * Compute $z = g^(l)(x)$, where $g^(l)()$ is the function whose input is $x$ and whose output is the layer $l$ output of the DNN. * Evaluate $f_Z|c^∗(z | c^∗)$ and declare an attack detection if this value is below a preset threshold.
Detecting Adversarial Samples from Artifacts · The Method from CITEMARK143Z · 0.81
Compares with 1 paper
-
This method significantly improves over the other ADA methods (excepting AWA-ADA-maxKL) and over [ref] , achieving 0.957 AUC (and a point with (TPR,FPR) = (0.91, 0.06)).
Detecting Adversarial Samples from Artifacts · Experiments Involving Several ADA Variants · 1.00
Cites as background 1 paper
-
Focusing on statistical classification, prominent attack types include: i) data poisoning attacks on training data, with the typical goal to degrade a learned classifier's accuracy, e.g. [ref] , [ref] , [ref] , but more recently to create “backdoors”, e.g. [ref] ; ii) reverse engineering attacks, which seek to learn a non-public (black box/undisclosed) classifier's decision-making rule by making numerous (even random) queries to the classifier [ref] , [ref] ; and, perhaps most importantly, iii) test-time evasion attacks [ref] , wherein test (operational) examples are perturbed human-imperceptibly, but in such a way that the classifier's decision is changed (and now disagrees with a consensus human decision), e.g. [ref] , [ref] , [ref] , [ref] , [ref] .
Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning · Introduction · 1.00
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.