REVIEW 4 major objections 5 minor 26 references
A New Kind of Adversarial Example: Measuring the Human-Model Gap, and Its Relationship to OOD Detection
T0 review · 4 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash
Pith's one-line read Large, visible perturbations that a human cannot recognize leave a model's correct prediction unchanged at 100% confidence, and standard confidence-based monitors fail to flag them.
desk verdict An honest and useful evaluation of a real failure mode, but the headline human–model gap leans on a proxy that is confounded with cross-model transfer failure; the OOD and adaptive-evasion results are the stronger contribution. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the NKE (New Kind of Adversarial Example), defined by a reversed perturbation constraint: instead of bounding the perturbation above by a small epsilon_s and requiring the prediction to change, one bounds it below by a large epsilon_l and requires the prediction to remain unchanged, D(x,x') > epsilon_l and f(x') = y. The attack is an iterative FGSM-style optimizer with an exterior projection that keeps the iterate on or outside the L2 ball of radius epsilon_l around the clean image. The evaluation machinery includes recognizability proxies (a second independently trained CNN, a classifier trained on Canny edge maps, and CLIP zero-shot), OOD detectors (maximum softmax pr
What would settle it
Run the released forced-choice harness with a properly powered sample (e.g., 50+ participants) on CIFAR-10 NKE images at the largest perturbation where the source model stays at 100% confidence. If human forced-choice accuracy does not fall to near chance (say, remains above 80%), while the model still labels them all correctly, the human–model gap claim collapses.
Extended reading notes
Core claim
The paper establishes three findings. First, the human–model gap is real on natural images: an independent generic recognizer falls to 48.5% at the largest perturbation while the source model stays at 100%, and this is not explained by signal loss because a matched-magnitude Gaussian control degrades recognizability even faster; a CLIP zero-shot proxy confirms the gap at ImageNet too. Second, NKE images are an OOD phenomenon invisible to confidence/energy/calibration monitoring (0% detection, ECE≈0) while a feature-space Mahalanobis detector flags 100%—but this detection is not adaptively robust, as an attacker who adds the Mahalanobis distance to the objective drives detection from 100% to
Load-bearing premise
The load-bearing premise is that a second network's accuracy on NKE images stands in for human recognition accuracy; the headline ~50-point gap could partly be cross-model transfer failure rather than loss of human-recognizable content.
Editorial extensions
If this is right
- Confidence-based safety monitoring is structurally blind to NKE: any system that gates on softmax confidence or energy scores should be audited against large-perturbation, same-label attacks.
- Feature-distance OOD detection (e.g., Mahalanobis) catches NKE when the attacker is oblivious, but it is not a robust defense once the attacker knows it is present, since a small penalty in the attack objective evades it completely.
- Classical adversarial robustness is orthogonal to NKE resistance: adversarial training, randomized smoothing, and input transformations leave NKE success near 100%, so robustness to small perturbations does not transfer to this failure mode.
- NKE transferability falls as task complexity rises (MNIST ≈0.97 → CIFAR-10 ≈0.36 → ImageNet ≈0.10), meaning on natural images the perturbation is highly model-specific and leaves little human-aligned signal.
- The texture-before-shape mechanistic dissociation suggests that the surviving edge skeleton is insufficient for recognition even for shape-biased readers, pointing to a deeper mismatch between model and human perceptual priors.
Reading between the lines
- The results imply that any deployment relying on max-softmax confidence as a proxy for input meaningfulness is vulnerable to a regime that is neither a small perturbation nor a natural OOD input, and existing monitors will not alert.
- Since the adaptive attacker need only know that a feature-distance detector is in use—not its parameters—the paper's finding generalizes beyond the specific Mahalanobis detector; our editorial take is that adaptively-robust detection requires construction that treats the detector as a known target, not post-hoc monitoring.
- The CLIP result at ImageNet (NKE less recognizable than equal-energy noise) suggests the perturbation actively destroys broad semantic content; this could be used as a stress test for foundation-model robustness, but the paper does not make that extension.
- The small human pilot (N=5) corroborates the gap but is underpowered for the signal-vs-structure contrast, so a properly powered crowd-sourced study using the released harness is the natural next test to confirm the proxy-based claim.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies 'opposite' adversarial examples (NKE): large, clearly visible perturbations that keep the model's original correct prediction while human recognizability drops. It makes three main claims: (1) the human–model gap is real, supported by a VGG-11 recognizer proxy, a matched-L2 Gaussian control, a 5-participant pilot, and a CLIP zero-shot proxy; (2) confidence- and energy-based OOD detectors and calibration are structurally blind to NKE, while a Mahalanobis feature-distance detector flags all such examples on ResNet-18 but is defeated by an adaptive attacker at no cost to success; and (3) classical defenses (adversarial training, randomized smoothing, input transformations) do not mitigate NKE, with resistance to small-epsilon attacks uncorrelated with resistance to large-epsilon_l attacks. The paper also reports a texture-vs-shape mechanism and a Vision Transformer probe. It includes baseline replication of prior work, explicit limitations, and a released code/human-study harness.
Significance. If the claims are established, the paper identifies a practically important failure mode: models can be confident and correct on inputs that humans cannot recognize, and the standard output-layer monitoring tools do not detect this. The adaptive-evasion result is a useful caution about feature-distance detectors. The paper has notable strengths: it ships reproducible code and a ready-to-run human-study harness; it checks several central measurements against external baselines; and it is unusually candid about its own limitations. However, the headline human–model gap currently rests on a proxy that is numerically confounded with known cross-model transfer collapse, and the only direct human evidence is a small pilot that cannot separate adversarial structure from generic signal loss. The CLIP evidence is stronger but appears only in an appendix and shows a scale-dependent reversal of the signal-loss comparison. These issues are substantial but addressable by reframing and by foregrounding the non-CNN proxy and pilot as the primary human evidence.
major comments (4)
- [Sec. 4.2, Table 1, Table 3] The headline '~50-point gap' is measured with a VGG-11 recognizer on ResNet-18 NKE. Table 1 reports VGG-11's black-box label-kept rate on ResNet-18 NKE as 0.50 at a comparable epsilon, and Table 3 reports the generic recognizer at 0.485 at epsilon_l=16. These are essentially the same number, so the gap may be the well-known cross-model transfer collapse rather than a human-recognition effect. The paper explicitly concedes this conflation at ImageNet (Sec. 4.9) but not on CIFAR-10. To make the central claim load-bearing, move the CLIP result (Sec. 4.10) into the main analysis and/or report a non-CNN proxy on CIFAR-10 in the main text; the current phrasing 'first quantitative (proxy) evidence for the gap' is not supported.
- [Abstract; Sec. 4.10 / Fig. B.12] The abstract states that 'a matched-magnitude Gaussian control degrades recognizability faster' as a general finding. At ImageNet the CLIP proxy shows the opposite ordering: NKE is less recognizable than equal-energy noise (0.33 vs 0.61 at epsilon_l=100). Thus the 'not signal loss' effect is scale-dependent. The abstract and Sec. 5 should state this as a CIFAR-10 result (and pilot-consistent but not statistically significant), not as a universal property.
- [Sec. 4.3; Sec. 4.6; Table C.14] The claim that NKE is 'visible only to feature-distance detection' is architecture-dependent. On ViT-sourced NKE, Mahalanobis detection peaks at 0.50 (Table C.14), and Sec. 4.6(iii) reports 43–50% on the ViT penultimate. The confidence/energy blindness is robust, but the 'only feature-distance detector sees it' claim is overstated. Please restrict the claim to the ResNet-18 setting or explain the architecture dependence explicitly.
- [Sec. 4.2, Table 2] The N=5 pilot has per-cell counts of 12–23 and bootstrap CIs that overlap substantially between the NKE and Gaussian-control conditions; the paper itself concedes that the signal-vs-structure contrast is 'not statistically separable at this sample size.' Since this is the only direct human evidence, the 'not signal loss' part of Contribution 1 relies entirely on the proxy. Either report the pilot as corroborating only the overall gap (not the structure specificity), or collect a properly powered sample before claiming the dissociation.
minor comments (5)
- [Sec. 3.1, Eq. (1)] The distance D(x,x') is not defined; the text later uses L2 norm. Define the norm in Eq. (1).
- [Sec. 4.7, Table 7] The text and Fig. 7 mention randomized smoothing, but Table 7 does not include a randomized-smoothing row. Reconcile the table and figure.
- [Abstract; Sec. 3.2] The abstract says 'ImageNet' while Sec. 3.2 and Sec. 4.9 clarify that the ImageNet-scale experiments use Imagenette (a 10-class subset). This should be stated in the abstract or at first use.
- [Data availability] The repository URL differs between the abstract (github.com/aliborji/NKE) and the end of Sec. 5 (github.com/alikayyam/NKE_attack.git). Harmonize the URLs.
- [Sec. 4.2] The statement that a human 'would no longer recognize' the image (Fig. 2 caption) is a strong claim; given the proxy-based evidence, hedge to 'the proxies and pilot suggest...'.
Circularity Check
Two headline 'predictions' reduce to construction choices: the ~50-point human gap is the known black-box transfer collapse relabeled, and the 0% MSP/energy/ECE blindness is entailed by the label-preserving attack objective.
-
renaming known result
[Sec. 4.2 (Table 3) vs. Sec. 4.1 (Table 1); Sec. 4.9]
"On CIFAR-10, the independent generic recognizer falls to 48.5% at ϵl=16 while the source model stays at 100% — a ~50-point gap, the first quantitative (proxy) evidence for the gap the original formulation assumed."
The 'generic recognizer' in Table 3 is VGG-11, and Table 1 reports VGG-11's label-kept rate on ResNet-18-sourced NKE as 0.50 at the second-largest grid level — numerically the same quantity as the 0.485 generic-recognizer accuracy at ϵl=16. The headline human–model gap is therefore not an independent human measurement but the already-known NKE black-box transfer collapse, relabeled as human recognizability. The paper itself concedes this in Sec. 4.9: 'The transfer curve doubles as the generic-recognizer human proxy' and 'The second-CNN proxy above conflates "a human would recognize this" with "this transfers to another CNN"'. This is a renaming of a known transferability result under new coordinates, not a derivation of a human gap.
-
fitted input called prediction
[Sec. 3.1 (attack objective), Sec. 3.3 (selection), Sec. 4.3 / Table 4]
"minimizing it, rather than maximizing it as in a classical attack, is what keeps the model’s prediction correct throughout the attack ... On CIFAR-10, MSP and energy detectors flag 0% of NKE images and ECE is ≈0 — the model is confident and correct, so calibration looks perfect on inputs no human could label."
NKE is defined by f(x′)=y (Eq. 1), the generator explicitly minimizes the true-label loss to keep the prediction correct, and the pipeline keeps only images 'still classified with the original label after, per Eq. 1'. Thus 100% model accuracy and high confidence are optimization/selection criteria, not empirical outcomes. MSP and energy are confidence-based scores, and ECE measures the accuracy/confidence match; with both accuracy and confidence forced high by construction, the reported 0% detection and ECE≈0 are entailed by the attack objective. The paper's own explanation — 'the model is confident and correct' — states the input that guarantees the blindness, so this result is a forced consequence of the construction rather than an independent discovery about OOD detectors.
full rationale
Two steps in the paper's claimed derivation chain reduce to their own inputs by construction. First, the headline CIFAR-10 'human–model gap' is measured with VGG-11 as the 'independent generic recognizer' on ResNet-18-generated NKE; that number (0.485 at ϵl=16) is essentially Table 1's black-box label-kept rate (0.50), i.e., the known NKE transfer collapse reproduced from Nie et al. The paper transparently admits the conflation in Sec. 4.9, but the quantitative gap presented as novel human evidence is a relabeling of that transfer result. Second, the MSP/energy 0% detection and ECE≈0 are not independent empirical findings: the attack is defined and optimized to preserve the true label and confidence, and the evaluation set is filtered to images that keep the original label, so output-confidence-based monitors are blind by construction. The paper's own phrase 'the model is confident and correct' names the premise that produces the result. The score is 6 rather than higher because the paper is not wholly circular: the matched-Gaussian control comparison, the CLIP proxy, the N=5 human pilot, the Mahalanobis/adaptive-attack experiments, and the defense-orthogonality measurements are external evaluations with real, non-reduced content. The self-citations ([3], [21]) are disclosed, are not used as the sole support for the central measurements, and are not a uniqueness or ansatz chain. The circularity is partial: two headline results are forced by the construction, while other contributions retain independent empirical content.
Assumptions & free parameters
free parameters (5)
- Perturbation magnitude grid (ϵl) =
1, 2.5, 5, 8, 12, 16 on CIFAR-10; up to 100 on ImageNet
- OOD detection operating point =
5% false-positive rate on held-out clean data
- Attention gate for pilot filtering =
0.55 clean-trial accuracy (CIFAR-10)
- Attack hyperparameters =
60 steps; momentum variants NMI-FGSM/NMI-FGM
- Canny thresholds and GLCM offset/feature choice =
not fully specified in the text
assumptions (7)
- domain assumption NKE attack machinery reliably produces images with D(x,x')>ϵl and f(x')=y at ~1.0 confidence
- domain assumption A second CNN (VGG-11), a Canny-shape CNN, and CLIP are valid computational proxies for human recognition
- domain assumption An L2-matched Gaussian control isolates perturbation structure from generic signal loss
- domain assumption ImageNet-scale results on Imagenette (10 classes) transfer to full ImageNet
- domain assumption GLCM cosine and Canny edge F1 faithfully measure texture and shape preservation
- domain assumption Four CIFAR-10 defense configurations suffice to estimate the small-ϵ vs large-ϵl robustness correlation
- standard math Iterative gradient descent on the classification loss finds non-trivial minimizers while the exterior projection holds
Cite this review
Pith. "Pith review of A New Kind of Adversarial Example: Measuring the Human-Model Gap, and Its Relationship to OOD Detection." pith.science (2026). https://pith.science/paper/SWZRYEQG
@misc{pith2026260722722,
author = {Pith},
title = {Pith review of: A New Kind of Adversarial Example: Measuring the Human-Model Gap, and Its Relationship to OOD Detection},
year = {2026},
howpublished = {\url{https://pith.science/paper/SWZRYEQG}},
note = {Machine review of arXiv:2607.22722}
}
read the original abstract
Almost all adversarial attacks add an imperceptible perturbation to fool a model. We instead study the opposite: a large, clearly visible perturbation that causes the model to keep its original, correct prediction, even though a human would no longer recognize the image. Prior work showed such examples can be generated at scale but left three questions untested: whether humans really perform worse than the model, whether standard out-of-distribution (OOD) detection and calibration tools catch it, and whether existing defenses mitigate it. We answer all three on MNIST, CIFAR-10, and ImageNet. (i) An independent recognizer proxy drops to ~49% on CIFAR-10 while the model stays at 100% -- a gap a small human pilot (N=5) corroborates directly and that is not explained by signal loss (a matched-magnitude Gaussian control degrades recognizability faster); a CLIP zero-shot proxy confirms the gap at ImageNet scale too. (ii) Confidence- and energy-based OOD detectors and calibration are structurally blind (0% detection, ECE ~= 0), while a feature-space Mahalanobis detector flags 100% -- but is evaded by an adaptive attacker at no cost to success. (iii) No classical defense, including adversarial training (45% robust accuracy), reduces attack success (correlation with large-epsilon_l resistance r ~= 0). A mechanistic analysis further shows the attack destroys low-level texture far faster than edge/shape structure.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
Szegedy, W
C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, R. Fergus, Intriguing properties of neural networks, in: International Confer- ence on Learning Representations (ICLR), 2014
2014
-
[2]
I. J. Goodfellow, J. Shlens, C. Szegedy, Explaining and harnessing adversarial examples, in: International Conference on Learning Representations (ICLR), 2015
2015
-
[3]
Borji, A new kind of adversarial example, arXiv preprint arXiv:2208.02430 (2022)
A. Borji, A new kind of adversarial example, arXiv preprint arXiv:2208.02430 (2022)
arXiv 2022
-
[4]
Nguyen, J
A. Nguyen, J. Yosinski, J. Clune, Deep neural networks are easily fooled: High confidence predictions for unrecognizable images, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, pp. 427–436
2015
-
[5]
X. Nie, G. Xiao, S. Pan, B. Wang, H. Ge, T. Fang, A new type of adversarial examples, arXiv preprint arXiv:2510.19347 (2025)
arXiv 2025
-
[6]
Kurakin, I
A. Kurakin, I. Goodfellow, S. Bengio, Adversarial machine learning at scale, in: International Conference on Learning Representations (ICLR), 2017
2017
-
[7]
Kurakin, I
A. Kurakin, I. Goodfellow, S. Bengio, Adversarial examples in the physical world, in: International Conference on Learning Representations (ICLR) Workshop, 2017
2017
-
[8]
Tramèr, A
F. Tramèr, A. Kurakin, N. Papernot, I. Goodfellow, D. Boneh, P. McDaniel, En- semble adversarial training: Attacks and defenses, in: International Conference on Learning Representations (ICLR), 2018. 29
2018
Show all 26 references
-
[9]
Carlini, D
N. Carlini, D. Wagner, Towards evaluating the robustness of neural networks, in: IEEE Symposium on Security and Privacy (S&P), 2017, pp. 39–57
2017
-
[10]
Y. Dong, F. Liao, T. Pang, H. Su, J. Zhu, X. Hu, J. Li, Boosting adversarial attacks with momentum, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 9185–9193
2018
-
[11]
Elsayed, S
G. Elsayed, S. Shankar, B. Cheung, N. Papernot, A. Kurakin, I. Goodfellow, J. Sohl-Dickstein, Adversarial examples that fool both computer vision and time-limited humans, in: Advances in Neural Information Processing Systems (NeurIPS), 2018
2018
-
[12]
Geirhos, C
R. Geirhos, C. R. Temme, J. Rauber, H. H. Schütt, M. Bethge, F. A. Wichmann, Generalisation in humans and deep neural networks, in: Advances in Neural Information Processing Systems (NeurIPS), 2018
2018
-
[13]
Z. Zhou, C. Firestone, Humans can decipher adversarial images, Nature Communications 10 (2019) 1334
2019
-
[14]
Hendrycks, K
D. Hendrycks, K. Gimpel, A baseline for detecting misclassified and out-of- distribution examples in neural networks, in: International Conference on Learning Representations (ICLR), 2017
2017
-
[15]
K. Lee, K. Lee, H. Lee, J. Shin, A simple unified framework for detecting out-of-distribution samples and adversarial attacks, in: Advances in Neural Information Processing Systems (NeurIPS), 2018
2018
-
[16]
W. Liu, X. Wang, J. Owens, Y. Li, Energy-based out-of-distribution detection, in: Advances in Neural Information Processing Systems (NeurIPS), 2020
2020
-
[17]
Ovadia, E
Y. Ovadia, E. Fertig, J. Ren, Z. Nado, D. Sculley, S. Nowozin, J. Dillon, B. Lakshminarayanan, J. Snoek, Can you trust your model’s uncertainty? 30 evaluating predictive uncertainty under dataset shift, in: Advances in Neural Information Processing Systems (NeurIPS), 2019
2019
-
[18]
Madry, A
A. Madry, A. Makelov, L. Schmidt, D. Tsipras, A. Vladu, Towards deep learning models resistant to adversarial attacks, in: International Conference on Learning Representations (ICLR), 2018
2018
-
[19]
Cohen, E
J. Cohen, E. Rosenfeld, Z. Kolter, Certified adversarial robustness via random- ized smoothing, in: International Conference on Machine Learning (ICML), 2019, pp. 1310–1320
2019
-
[20]
C. Guo, M. Rana, M. Cisse, L. van der Maaten, Countering adversarial images using input transformations, in: International Conference on Learning Representations (ICLR), 2018
2018
-
[21]
Borji, Addressing the topological defects of disentanglement and adversarial robustness: notes on human vision, arXiv preprint arXiv:2208.11580 (2022)
A. Borji, Addressing the topological defects of disentanglement and adversarial robustness: notes on human vision, arXiv preprint arXiv:2208.11580 (2022)
2022 arXiv
-
[22]
Geirhos, P
R. Geirhos, P. Rubisch, C. Michaelis, M. Bethge, F. A. Wichmann, W. Bren- del, Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness, in: International Conference on Learning Representations (ICLR), 2018
2018
-
[23]
Y. Sun, Y. Ming, X. Zhu, Y. Li, Out-of-distribution detection with deep nearest neighbors, in: International Conference on Machine Learning (ICML), 2022
2022
-
[24]
Naseer, K
M. Naseer, K. Ranasinghe, S. Khan, M. Hayat, F. S. Khan, M.-H. Yang, Intriguing properties of vision transformers, in: Advances in Neural Information Processing Systems (NeurIPS), 2021. 31
2021
-
[25]
S. Tuli, I. Dasgupta, E. Grant, T. L. Griffiths, Are convolutional neural networks or transformers more like human vision?, in: Proceedings of the Annual Meeting of the Cognitive Science Society (CogSci), 2021
2021
-
[26]
Dosovitskiy, L
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Un- terthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, N. Houlsby, An image is worth 16x16 words: Transformers for image recog- nition at scale, in: International Conference on Learning ...
2021
Reviewed August 1, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.