REVIEW 4 major objections 5 minor 1 cited by
Neural Networks Use Distance Metrics
T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read This paper argues that trained neural networks with ReLU and Absolute Value activations encode inputs as distances to learned decision boundaries rather than as feature strengths, and supports the claim with perturbation experiments on…
desk verdict A readable, honestly-written paper whose central claim fails: the "distance" perturbation is just a decision-boundary shift, so the headline result is not established. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying mechanism is the perturbation layer inserted between the first linear layer and the activation function, which applies $y = \min(\text{scale} \cdot x + \text{offset}, \text{clip})$ to each unit with three fixed parameters. Offset moves the pre-activation decision boundary by a percentage of the unit's output range; scale multiplies the activation magnitudes; clip caps them. Comparing accuracy across the three perturbations is the experiment's probe for whether the network depends on distance-to-boundary or on activation magnitude. The theoretical object behind the probe is the Mahalanobis distance, a data-covariance-aware distance measure, which the companion theory says a linear node's pre-activation projection computes.
What would settle it
Compensate for an offset perturbation by adding the corresponding constant to the second linear layer's bias so the post-activation decision boundary stays fixed; if accuracy is fully restored, the collapse under offset is boundary movement rather than learned distance. A second test: find a trained ReLU or Abs network whose accuracy collapses under large scaling or clipping while surviving 1% offsets, which would break the claimed asymmetry.
Extended reading notes
Core claim
On its own terms, the paper's discovery is that ReLU and Absolute Value networks behave as distance-measuring machines: a linear node projects the input, and its activation records how far that projection sits from a learned boundary, so the smallest post-activation values mark the strongest match to the node's feature. Empirically, offset perturbations that move the boundary by 1–3% of the activation range degrade accuracy sharply, whereas scaling by up to 1000% and clipping down to 50% of the maximum leave performance nearly intact. The Absolute Value network is more sensitive because folding the feature space makes it select a single minimum-distance feature, while ReLU selects a set of accepted features and degrades more gradually. This pattern is presented as empirical validation of the Mahalanobis-distance interpretation of linear nodes.
Load-bearing premise
The load-bearing premise is that an offset added before ReLU or Abs is a distance-only perturbation that leaves intensity features intact, but that offset is exactly a change to the first layer's bias—a decision-boundary shift any linear classifier depends on—and the paper offers no formal definition of intensity to rule out a symmetric control.
Editorial extensions
If this is right
- Activation magnitude should not be read as feature strength; saliency and feature-visualization methods that emphasize large activations may be highlighting the values the classifier cares least about.
- Adversarial vulnerability is a natural consequence, since tiny input changes that shift pre-activation projections by a few percent can move the network's selected features across decision boundaries.
- Absolute Value networks rely on a single precise distance measurement, so they should be expected to degrade faster under small boundary shifts than ReLU networks, which can fall back on alternative accepted features.
- Interpretability and robustness work should be redirected toward the geometry of decision boundaries and small-magnitude activations rather than toward suppressing or amplifying large activations.
Reading between the lines
- An offset before ReLU or Abs is mathematically equivalent to changing the first layer's bias, so offset sensitivity alone does not separate the distance-metric hypothesis from the generic fact that linear classifiers depend on their decision boundaries; a cleaner control would keep the boundary fixed while varying distances.
- A testable extension: train the same architecture on synthetic data with known covariance structure and check whether the first-layer weight vectors align with the inverse covariance matrix implied by the Mahalanobis distance.
- If the distance-based reading is right, pruning and quantization schemes that discard small activations may be removing the most decision-relevant signal, so low-magnitude activations near boundaries should be protected rather than discarded.
- The scaling invariance could be tested against the LogSoftmax explanation by replacing the cross-entropy head with a linear head and repeating the scale perturbation; if invariance disappears, normalization is the cause, and if it persists, distance-based computation is.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents perturbation experiments on a two-layer MLP with ReLU or absolute-value activations trained on MNIST. A custom perturbation layer applies scale, offset, and clipping to the first-layer pre-activations. The authors label offset shifts as 'distance-based' perturbations and scaling/clipping as 'intensity-based' perturbations, and report that accuracy is sensitive to offsets but robust to scaling/clipping. They conclude that the networks learn Mahalanobis-distance representations and that this validates the author's prior theoretical framework [Oursland, 2024].
Significance. If the conclusion were valid, it would be a notable challenge to the default intensity/threshold reading of activations and would give empirical traction to a Mahalanobis-distance account of linear layers. The paper is transparent about its implementation and provides code, training details, 20 repeated runs, and full result tables. However, the experiments as designed do not isolate distance-to-prototype computation from ordinary decision-boundary sensitivity, and the intensity control is not independently defined. The positive contribution is therefore an interesting perturbation setup and an observation about scaling invariance, not evidence for the paper's central claim.
major comments (4)
- [Section 4.1, Eq. (1), Figure 1] The 'distance-based perturbation' is an additive offset applied to the pre-activation z = Wx + b before the nonlinearity (y = min(scale * x + offset, clip)). For fixed scale, this is exactly a change of the bias/threshold: z + offset = Wx + (b + offset). Section 6.1 and Figure 3 confirm that offset perturbations change the set of accepted features by moving the ReLU threshold or the Abs fold point. Any threshold-based or intensity-based classifier is sensitive to a movement of its decision boundary, so the observed accuracy drop under offset does not discriminate between distance-based and intensity-based representations. The central contrast therefore rests on a perturbation that is not a diagnostic of distance metrics.
- [Section 6.4] The paper concedes that there is no formal definition of an intensity feature: 'we cannot definitively disprove intensity-based interpretations due to the lack of a widely accepted definition of what constitutes an intensity feature' and 'we find no equivalent statistical measure for intensity.' Without such a definition, classifying scaling and clipping as intensity perturbations is not justified, and the claim that intensity features are 'maintained' or 'targeted' is untestable. The asymmetry between the two perturbation families is assumed rather than derived, which undermines the central comparison.
- [Section 5, Tables 2 and 3] The text states that intensity perturbations produced fluctuations that were 'not statistically significant (p > 0.05), as detailed in Table 2 and Table 3.' This is contradicted by the tables: Table 3 reports P-values on the order of 1e-16 for every cutoff level, and Table 2 reports P < 0.05 for scale = 1%, 5%, and 10%, with ReLU accuracy falling to 75.33% at scale = 1%. The claimed robustness to intensity perturbations is therefore not supported by the paper's own statistics.
- [Sections 3, 4.1, 6.4, and 7] The experiment is framed as validation of [Oursland, 2024], and the same framework is used to label offset perturbations as 'distance-based' (Section 4.1) and to interpret the results (Sections 6.1 and 6.4). This is circular: the perturbation classification is derived from the theory being tested, and the outcome is then read as confirming that theory. An independent operationalization of 'distance-to-prototype' (for example, perturbing estimated prototype locations or covariance structures rather than adding a bias shift) would be needed to break the circularity.
minor comments (5)
- [Section 2] The text refers to 'McCulloch and Pitt' but the reference list correctly gives 'McCulloch and Pitts'; please correct the in-text name.
- [Section 4.1] The perturbation formula {scale = (1 - p) * r, offset = p * r} is inconsistent with the stated training-time identity scale = 1, offset = 0; please clarify how r is computed and how p = 0 yields the baseline operation.
- [Section 5] The qualitative description 'approximately 98% for ReLU and 99% for Abs' should be replaced by the precise baseline values from Table 1.
- [Appendix A, Tables 2-4] The t-statistics and p-values are reported with very few significant digits and with negative t-statistics for two-sided tests; please state the null hypothesis and the test procedure (paired vs. independent, one-sided vs. two-sided) explicitly.
- [Section 6.4] The term 'principal disponent' is introduced without definition, and the paragraph appears to assert the non-existence of the entity it names; please define or remove this term.
Circularity Check
The 'distance perturbation' is a bias shift imported from the author's own framework, so the experiment's positive result is partly built into its labels.
-
self definitional
[Section 4.1, Perturbation Design]
"Distance-based features are expected to lie near the decision boundary. By shifting the decision boundary, we increase the distance between active features and the boundary. If these features are critical for classification, this shift should result in reduced model performance. For each node, we calculate the output range, scale by the specified percentage, and then apply the offset as a percentage of the range. The perturbation equation for a given percentage p and range r is: {scale = (1 − p) · r, offset= p · r}."
The distance perturbation reduces to a bias shift: y = min(scale·x + offset, clip) with scale=(1−p)r and offset=pr is an affine change to the pre-activation of Linear(784), exactly what changing that layer's bias would do. ReLU and Abs then move their threshold/fold point, changing the active feature set, which Section 6.1 confirms ('offset perturbations modify the set of accepted features'). Any threshold-based (intensity) classifier depends on that same boundary. Therefore sensitivity to offset does not specifically implicate a learned Mahalanobis metric; the positive result is baked into the theory-laden definition of 'distance perturbation'.
-
self citation load bearing
[Section 3 Background and Section 6.4 The Problem with Intensity]
"In our previous work, Interpreting Neural Networks through Mahalanobis Distance, we established a mathematical link between linear nodes with absolute value activation functions and statistical distance metrics. [Oursland, 2024] This framework suggests that neural networks may naturally learn to measure distances rather than intensities. ... The distance features in the network are easily explained as a Mahalanobis distance of a principal component as described in [Oursland, 2024]."
The paper's hypothesis and its interpretive vocabulary both come from the same self-cited preprint. The perturbation labels ('distance' vs 'intensity') are taken from that framework, and the observed pre-activation structure is 'easily explained' only by citing that same work. The conclusion states that the paper 'provides empirical validation for the theoretical connection between neural networks and Mahalanobis distance proposed in [Oursland, 2024]', so the experiment validates the author's own prior theory using categories that prior theory supplies. Since the cited work is not machine-checked and no external operationalization of 'Mahalanobis distance feature' is given, the self-citation is load-bearing rather than independent confirmation.
1 more flagged steps
-
renaming known result
[Section 6.4, The Problem with Intensity]
"we cannot definitively disprove intensity-based interpretations due to the lack of a widely accepted definition of what constitutes an intensity feature. ... we find no equivalent statistical measure for intensity that can be expressed through a linear equation. ... If we propose that relative values between nodes preserve intensity information, this begins to sound suspiciously like a distance metric."
The intensity side of the control is undefined, so the contrast is unfalsifiable: any robustness result can be attributed to choosing the 'wrong' intensity perturbation. When scaling invariance is observed, the paper relabels hypothetical intensity-preserving relative values as 'suspiciously like a distance metric', converting a possible intensity explanation into evidence for distance. This is renaming rather than testing: the known scale-invariance of the classifier's decision under LogSoftmax/linear scores is re-presented as a discovery about distance metrics, with no independent definition of either construct.
full rationale
The measurements themselves are real and are reported with baselines and statistics, so this is not data-level circularity. The circularity is in construct validity: Section 4.1 defines the distance perturbation as a boundary shift (an affine/bias change), and Section 5's conclusion reads off sensitivity to that boundary shift as evidence for Mahalanobis-distance representations. The author's prior work [Oursland 2024] provides both the theoretical claim and the 'distance' label used to interpret the features, and Section 6.4 admits there is no formal definition of intensity, making the control condition unable to falsify the hypothesis. Consequently the central empirical contrast is partially forced by the paper's own definitions and self-citation chain, though some independent empirical content remains (e.g., the differing sensitivity of Abs vs ReLU and the quantitative perturbation tables).
Assumptions & free parameters
assumptions (4)
- domain assumption Trained single-hidden-layer MLPs on MNIST are representative of neural networks in general.
- ad hoc to paper The Mahalanobis-distance framework from Oursland (2024) correctly identifies distance-based features.
- ad hoc to paper Offset, scaling, and clipping perturbations respectively isolate distance and intensity properties.
- domain assumption Training-set accuracy is an appropriate measure of feature reliance.
invented entities (1)
-
principal disponent
Cite this review
Pith. "Pith review of Neural Networks Use Distance Metrics." pith.science (2026). https://pith.science/paper/2TT3Y5HS
@misc{pith2026241117932,
author = {Pith},
title = {Pith review of: Neural Networks Use Distance Metrics},
year = {2026},
howpublished = {\url{https://pith.science/paper/2TT3Y5HS}},
note = {Machine review of arXiv:2411.17932}
}
read the original abstract
We present empirical evidence that neural networks with ReLU and Absolute Value activations learn distance-based representations. We independently manipulate both distance and intensity properties of internal activations in trained models, finding that both architectures are highly sensitive to small distance-based perturbations while maintaining robust performance under large intensity-based perturbations. These findings challenge the prevailing intensity-based interpretation of neural network activations and offer new insights into their learning and decision-making processes.
Figures
Forward citations
Cited by 1 Pith paper
-
Neural Networks Learn Distance Metrics
A two-layer MNIST study argues that networks prefer distance-based representations, but the evidence is mostly architecture-specific and the OffsetL2 layer resembles known RBF units.
Reference graph
Works this paper leans on
-
[1]
Neural machine translation by jointly learning to align and translate
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014
arXiv 2014
-
[2]
John S Bridle. Probabilistic interpretation of feedforward classification network outputs, with relationships to statistical pattern recognition. Neurocomputing, 3 0 (1): 0 227--236, 1990
work page 1990
-
[3]
a ckinger, and Roopak Shah. Signature verification using a
Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard S \"a ckinger, and Roopak Shah. Signature verification using a" siamese" time delay neural network. In Advances in neural information processing systems, pages 737--744, 1994
work page 1994
-
[4]
Radial basis functions, multi-variable functional interpolation and adaptive networks
David S Broomhead and David Lowe. Radial basis functions, multi-variable functional interpolation and adaptive networks. Royal Signals and Radar Establishment Malvern (United Kingdom), 1988
1988
-
[5]
Michel Marie Deza and Elena Deza. Encyclopedia of distances. Springer, 2009
work page 2009
-
[6]
Visualizing higher-layer features of a deep network
Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of a deep network. University of Montreal, 1341 0 (3): 0 1, 2009
2009
-
[7]
Deep sparse rectifier neural networks
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 315--323, 2011
2011
-
[8]
Explaining and harnessing adversarial examples
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014
arXiv 2014
Show all 32 references
-
[9]
Multilayer feedforward networks are universal approximators
Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural networks, 2 0 (5): 0 359--366, 1989
1989
-
[10]
Backpropagation applied to handwritten zip code recognition
Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 1 0 (4): 0 541--551, 1989
1989
-
[11]
Gradient-based learning applied to document recognition
Yann LeCun, L \'e on Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86 0 (11): 0 2278--2324, 1998
1998
-
[12]
The mythos of model interpretability
Zachary C Lipton. The mythos of model interpretability. Queue, 16 0 (3): 0 31--57, 2018
2018
-
[13]
Towards deep learning models resistant to adversarial attacks
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083, 2017
2017 arXiv
-
[14]
On the generalized distance in statistics
Prasanta Chandra Mahalanobis. On the generalized distance in statistics. Proceedings of the National Institute of Sciences of India, 2 0 (1): 0 49--55, 1936
1936
-
[15]
A logical calculus of the ideas immanent in nervous activity
Warren S McCulloch and Walter Pitts. A logical calculus of the ideas immanent in nervous activity. The bulletin of mathematical biophysics, 5 0 (4): 0 115--133, 1943
1943
-
[16]
Methods for interpreting and understanding deep neural networks
Gr \'e goire Montavon, Wojciech Samek, and Klaus-Robert M \"u ller. Methods for interpreting and understanding deep neural networks. Digital Signal Processing, 73: 0 1--15, 2018
2018
-
[17]
Rectified linear units improve restricted boltzmann machines
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. Proceedings of the 27th international conference on machine learning (ICML-10), pages 807--814, 2010
2010
-
[18]
Feature visualization
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2 0 (11): 0 e7, 2017
2017
-
[19]
Interpreting neural networks through mahalanobis distance
Alan Oursland. Interpreting neural networks through mahalanobis distance. arXiv preprint arXiv:2410.19352, 2024
2024 arXiv
-
[20]
The perceptron: A perceiving and recognizing automaton
Frank Rosenblatt. The perceptron: A perceiving and recognizing automaton. Technical Report 85-460-1, Cornell Aeronautical Laboratory, 1957
1957
-
[21]
The perceptron: a probabilistic model for information storage and organization in the brain
Frank Rosenblatt. The perceptron: a probabilistic model for information storage and organization in the brain. Psychological review, 65 0 (6): 0 386, 1958
1958
-
[22]
Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead
Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1 0 (5): 0 206--215, 2019
2019
-
[23]
Learning representations by back-propagating errors
David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back-propagating errors. nature, 323 0 (6088): 0 533--536, 1986
1986
-
[24]
Explainable ai: interpreting, explaining and visualizing deep learning
Wojciech Samek, Gr \'e goire Montavon, Andrea Vedaldi, Lars Kai Hansen, and Klaus-Robert M \"u ller. Explainable ai: interpreting, explaining and visualizing deep learning. Springer, 2019
2019
-
[25]
Deep learning in neural networks: An overview
J \"u rgen Schmidhuber. Deep learning in neural networks: An overview. Neural networks, 61: 0 85--117, 2015
2015
-
[26]
Facenet: A unified embedding for face recognition and clustering
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 815--823, 2015
2015
-
[27]
Deep inside convolutional networks: Visualising image classification models and saliency maps
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034, 2013
2013 arXiv
-
[28]
Intriguing properties of neural networks
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013
2013 arXiv
-
[29]
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017
2017
-
[30]
Understanding neural networks through deep visualization
Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson. Understanding neural networks through deep visualization. In International conference on machine learning, pages 1576--1585. PMLR, 2015
2015
-
[31]
Visualizing and understanding convolutional networks
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. European conference on computer vision, pages 818--833, 2014
2014
-
[32]
Learning deep features for discriminative localization
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2921--2929, 2016
2016
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.