{"id":"9eadaaba-d436-42da-b9ca-0598fa084f2a","arxiv_id":"2502.02103","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A two-layer MNIST study argues that networks prefer distance-based representations, but the evidence is mostly architecture-specific and the OffsetL2 layer resembles known RBF units.","lead":"This paper tests whether neural networks encode inputs as distances to learned prototypes rather than as activation intensities, using six small MNIST architectures. It introduces OffsetL2, a radial-basis-like distance layer, and reports that this layer reaches about 97.6% accuracy with low variance.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The ReLU2/ReLU2-Neg contrast does not isolate 'distance bias': it only tests which side of the final ReLU kink the correct class must occupy, so the central claim is not supported by the experiments.","rationale":"I agree with the reader's weakest assumption: the architecture comparisons do not force intensity versus distance representations, and the ReLU2 collapse is confounded by the dead-ReLU optimization problem. The sign flip in ReLU2-Neg changes which side of the ReLU kink the correct class must occupy, so the performance difference can be explained without invoking any natural distance bias. A LeakyReLU control directly removes this confound. The OffsetL2 results are a valid empirical datapoint, but OffsetL2 is an RBF-like layer whose strong MNIST performance is expected and does not validate that standard ReLU networks prefer distances. Since the central claim is not established by the current experiments, the REJECT verdict remains appropriate; my stress-test does not change it.","tokens_in":9915,"tokens_out":5278,"duration_ms":54133,"concrete_test":"Train the same ReLU2 and ReLU2-Neg architectures with the final ReLU replaced by LeakyReLU(0.1), under the identical no-bias final layer, 5000-epoch SGD protocol, and 20 seeds. If ReLU2-Leaky recovers to within one point of ReLU2-Neg accuracy, then the collapse is caused by dead-ReLU gradient starvation rather than by any intensity constraint or distance bias, removing the paper's main empirical support for its central claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing evidence is the ReLU2 collapse vs ReLU2-Neg recovery (Sections 5.2-5.3). But the two architectures differ only by an output sign flip after the same final ReLU. With CrossEntropyLoss and no final bias (Section 4.2), ReLU2 must make the correct logit max(0, W_i h) positive and largest, which is fragile under random initialization because many preactivations are negative and the ReLU zeroes both output and gradient. ReLU2-Neg flips the sign to -max(0, W_i h), letting the correct class sit at the ReLU zero plateau while non-target classes are driven positive; this is a standard gradient-flow and optimization asymmetry, not evidence of a distance representation. Section 6.2 itself attributes ReLU2's failure to 86.5% of output nodes being dead or nearly dead, which is the dead-ReLU pathology, not evidence that the network prefers distances. No experiment probes whether learned representations are actually distance-based (e.g., whether logits correlate with Mahalanobis distance to prototypes); the representational type is inferred from the sign convention the authors imposed. Hence the central claim is unestablished, and Abs2-Neg's contradicting underperformance (Section 5.3) is explained post hoc rather than predicted.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript hypothesizes that neural networks have a natural bias toward distance-based representations, in which small activations indicate proximity to learned prototypes, and contrasts this with intensity-based representations in which large activations indicate feature presence. To test this, it trains simple two-layer MNIST models with six primary architectures (Abs, ReLU, Abs2, Abs2-Neg, ReLU2, ReLU2-Neg) and reports that ReLU2 collapses while ReLU2-Neg recovers, and that Abs2 retains baseline accuracy while Abs2-Neg degrades. It then introduces a geometric framework based on Mahalanobis distance and prototype/anti-prototype centers, and proposes the OffsetL2 layer, which explicitly computes a weighted L2 distance to learned reference points. The paper claims that the experiments validate the framework and that the OffsetL2 results support a distance-based view of neural representations.","tokens_in":10191,"tokens_out":3986,"duration_ms":38223,"significance":"If the central claim were established, the paper would offer a substantive interpretive shift for deep learning: reading activations as distances rather than intensities would affect architecture design and interpretability. The manuscript also includes a useful empirical artifact: a simple, reproducible two-layer setup, source code, and an explicit OffsetL2 architecture with strong MNIST accuracy and low variance. However, the evidence presented does not support the central claim. The ReLU2/ReLU2-Neg contrast is confounded by well-known dead-ReLU optimization behavior; the representational type is never measured directly; and the OffsetL2 validation is partly circular because that layer computes a distance by construction. The statistical reporting also contains inconsistencies that prevent the results from being interpreted as stated.","major_comments":[{"comment":"The ReLU2 versus ReLU2-Neg contrast does not isolate distance-based versus intensity-based representations. With CrossEntropyLoss and no final bias, ReLU2 must make the correct-class preactivation positive after the final ReLU, while ReLU2-Neg can place the correct class on the ReLU zero plateau and drive non-target classes positive. The dead-node statistics reported in §6.2 (33.0% permanently inactive, 53.5% active on less than 5% of inputs) are the standard dead-ReLU gradient-flow failure, and the discussion explicitly invokes this known mechanism. The experiment therefore cannot discriminate the paper's distance-bias hypothesis from ordinary optimization pathology.","section":"§5.2–5.3 and §6.2"},{"comment":"The representational type is assigned by the sign convention the authors impose, not measured in the trained networks. No analysis shows that ReLU2-Neg's output-layer geometry is aligned with class-conditional prototype centers, nor that the logits correlate with a Mahalanobis-distance-like quantity. The Abs2-Neg performance drop is explained post hoc with the unmeasured entities zc and z¬c, so the explanation has no predictive content as tested. A direct probe of the learned representation is needed before the central claim can be accepted.","section":"§6.1–6.3"},{"comment":"The OffsetL2 validation is partly by construction: the layer explicitly computes ||αi ⊙ (x − µi)||2, so its strong performance is expected for an RBF-like layer and cannot demonstrate that ordinary networks naturally prefer distance representations. In addition, the comparison mixes training conventions: the baselines and primary variants are trained for 5,000 epochs, while the OffsetL2 models are trained for 50,000 epochs, making the accuracy gains in Table 5 relative to Tables 2–3 uninterpretable as evidence of architectural advantage.","section":"§6.4 and Table 5"},{"comment":"There are multiple statistical reporting inconsistencies. The same t-statistic (t(38) = -17.33) is reported for both ReLU2's degradation and ReLU2-Neg's recovery; the comparison of ReLU2-Neg to baseline ReLU reports t(38) = -12.78 with p < 0.001 yet describes the two as 'statistically comparable'; and §6.2 gives ReLU2 accuracy as 47.20% ± 12.00% while Table 2 reports 56.31 ± 19.31. These discrepancies must be resolved before the empirical claims can be evaluated.","section":"§5.2–5.3 and §6.2"}],"minor_comments":[{"comment":"Figure 1 is referenced in the text and has a caption, but no figure appears in the manuscript.","section":"§6.1"},{"comment":"Model naming is inconsistent: 'ReLU2 Neg' and 'Abs2 Neg Bias' appear in Table 3, while 'ReLU2-Neg' and 'Abs2-Neg' are used elsewhere; the hyphenation and bias notation should be uniform.","section":"Tables 3 and 5"},{"comment":"The abstract emphasizes six architectural variants, but §6.4 introduces four additional OffsetL2-based models; the paper should clarify the relationship between the primary six and the later models.","section":"Abstract and §6.4"},{"comment":"The text contains 'p ¡ 0.001' instead of 'p < 0.001', and one t-test comparison appears to be copied from the previous paragraph.","section":"§5.3"}],"recommendation":"reject","confidential_remarks":"For the editor: the core distance-versus-intensity framework is largely drawn from two self-cited arXiv preprints, and the manuscript does not clearly separate assumptions inherited from those preprints from contributions established in this paper. The main issue, however, is internal: the key experiment is confounded, the representational type is never directly probed, and the validation is partly circular. These are load-bearing problems that would require a substantially different experimental design to fix."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis paper gives you a neat negative result—ReLU2 collapses on MNIST while ReLU2-Neg recovers—and a tidy empirical datapoint: OffsetL2, a weighted L2 prototype layer, hits 97.6% with 0.07% variance. That part is worth a look. The author is honest about the surprising Abs2-Neg degradation and tries to explain it with a geometric story. The writing is clear.\n\nBut the central claim—that networks naturally prefer distance-based representations—is not supported. The ReLU2/ReLU2-Neg contrast only flips the sign after the same ReLU: with CrossEntropyLoss and no bias, ReLU2 must push the correct class positive, so the optimizer drives most preactivations negative and kills the ReLU. ReLU2-Neg lets the correct class sit at zero while pushing others positive. That is a standard dead-ReLU optimization asymmetry, not evidence of a distance bias. The author's own Section 6.2 says the failure is due to 86.5% dead or nearly-dead output nodes. No experiment probes whether the learned features actually correspond to distances; the representational type is inferred from the imposed sign convention.\n\nThe statistical reporting has errors—t-statistics are repeated in places, and accuracy numbers differ between Table 2 and Section 6.2. That needs correction.\n\nOffsetL2 is conceptually a weighted L2/RBF layer with learned prototypes, which is known to do well on MNIST; the Mahalanobis framing is inherited from the author's prior preprints. So the novelty is modest.\n\nIf you work on distance-based layers, this paper gives you a clean baseline comparison. It deserves a serious referee, but with the understanding that major revisions would be required—the author should at least probe representations directly and fix the reporting errors.\n\nRecommendation: send it to peer review, but do not expect it to be accepted in its current form.","headline":"A clean empirical datapoint about an RBF-style layer, but the headline claim about distance bias is not established; the ReLU2 failure is a dead-ReLU artifact.","tokens_in":10744,"tokens_out":2957,"would_cite":false,"duration_ms":26359,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that neural networks naturally favor distance-based representations—smaller activations mean closer to a learned prototype—and shows that small architectural changes that force intensity-style outputs can collapse…","keywords":["distance-based representations","intensity-based representations","Mahalanobis distance","ReLU dead neurons","prototype learning","neural network interpretability","MNIST","OffsetL2"],"falsifier":"Train a ReLU2-Neg network on MNIST, identify each class prototype by intersecting the first-layer hyperplanes, and measure the correlation between an input's distance to the class-c prototype and the class-c logit. The paper's claim predicts that smaller distances correspond to larger (less negative) or near-zero logits in a consistent ordering across all ten classes; if the logits instead vary with intensity-style separation and show no distance ordering, the distance-representation reading is falsified.","tokens_in":9664,"feed_emoji":"📏","tokens_out":6209,"duration_ms":55023,"temperature":0.7,"pith_summary":"The paper sets out to establish that neural networks do not primarily encode features by activation magnitude (intensity), but by distance to learned prototypes: smaller activations signal proximity. It tests this by building six simple MNIST classifiers whose final layer is architecturally forced into one mode or the other, using a second activation for non-negativity and a negation layer to flip a distance code into an intensity code. The key empirical pattern is that ReLU2, forced into intensity mode, collapses to 56% accuracy with mass node death, while ReLU2-Neg, which can internally use distance, recovers to 96%; Abs2 stays at baseline but Abs2-Neg degrades. The paper then proposes OffsetL2, a layer that computes a weighted L2 distance to a learned per-class prototype, which reaches 97.6% with very low variance and closes the gap between normal and negated variants. If correct, the results imply that activation values in trained networks should be read geometrically, as distances, and that architecture design should accommodate distance-based representations rather than only intensity-based ones.","feed_headline":"Networks learn distances, not intensities—a second ReLU proves it","feed_subtitle":"Forcing intensity-style outputs collapses ReLU networks; adding a negation restores them, and explicit prototype-distance layers win.","key_machinery":"The carrying object is the distinction between distance and intensity representations, formalized through Mahalanobis distance. A distance representation is modeled as $f(x) = |W(x-\\mu)|_p$: a linear layer learns the eigenvector and scaling transform, the bias learns the prototype offset, and an Abs or ReLU pair encodes absolute distance. CrossEntropyLoss is the constraint that forces intensity output—high activation for the correct class—and a negation layer is the mechanism that flips a positive distance code into the required negative intensity code. The geometric picture is that each output hyperplane intersects the latent space at learned prototypes; a distance-mode hyperplane passes near the optimal class center $z_c$, while an intensity-mode hyperplane passes near anti-centers $z_{\\neg c}$. OffsetL2, $y_i = \\|\\alpha_i \\odot (x - \\mu_i)\\|_2$, carries the argument's final validation by making a single learned prototype per class explicit.","core_discovery":"On the paper's own terms, the central discovery is that ReLU-based architectures exhibit a natural bias toward distance-based learning. Cross-entropy loss with a non-negative final layer forces an intensity readout—large logits for the correct class—so a second ReLU creates an intensity-constrained network; that network fails catastrophically because minimizing activations for the 90% of examples that are not the target class drives nearly all pre-activations negative and kills 33% of output nodes outright. Adding a negation layer converts a positive distance code into a negative intensity code, restoring near-baseline performance. Abs networks behave differently because absolute value never kills nodes, but Abs2-Neg underperforms, which the paper traces to a geometric constraint: a hyperplane must pass through one optimal prototype point plus 127 less-optimal points, whereas Abs2 can choose among combinatorial anti-prototype configurations. The paper's final validation is OffsetL2, which explicitly models the distance to a single learned prototype per class and achieves the best accuracy and stability, supporting the claim that distance-based representations are the natural mode of neural computation.","pith_inferences":["Editorial inference: if the distance-bias claim generalizes, widely used practices that implicitly penalize or suppress activations—such as weight decay, dropout, or activation-based pruning—may be operating as prototype-distance regularizers and should be reinterpreted accordingly.","Editorial inference: the dead-node mechanism predicts that any clustered dataset with many non-target classes per decision, not just MNIST, should show a ReLU2-style collapse under intensity constraints; running the same six architectures on CIFAR-10 or a synthetic Gaussian-mixture task would test the scope.","Editorial inference: the OffsetL2 success suggests a direct architectural recipe—replace the final linear layer with a learned per-class Mahalanobis layer—that could be tested on larger-scale classification and in the hidden layers of deeper networks, not just the two-layer setting studied here."],"forward_implications":["In a trained network, a small activation of a node is evidence that the input is near that node's learned prototype, not evidence that the feature is absent or weak.","Putting a second ReLU immediately before logits—an intensity-forcing arrangement—can cause catastrophic dying-neuron collapse; a negation layer ahead of the logits can recover performance by allowing distance-based internal codes.","Abs-based layers are more robust under intensity constraints because no nodes die, but their distance-constrained variant pays a geometric price in high-dimensional prototype selection.","Explicit prototype-distance layers like OffsetL2 outperform implicit hyperplane-based prototype discovery on MNIST and sharply reduce run-to-run variance, with standard deviation down to 0.07%."],"supporting_citations":[{"why":"Supplies the Mahalanobis-distance interpretation of linear layers with Abs activations that the present framework extends.","marker":"Oursland [2024a]"},{"why":"Provides prior empirical evidence that networks respond to perturbations of distance relationships, motivating the distance-bias hypothesis.","marker":"Oursland [2024b]"},{"why":"Defines the Mahalanobis distance, the statistical metric the paper uses as the foundation for distance-based representations.","marker":"Mahalanobis [1936]"},{"why":"Provides the MNIST dataset and standard experimental setting used for all six architecture comparisons and OffsetL2 validation.","marker":"LeCun et al. [1998]"},{"why":"Documents the dying-ReLU phenomenon that the paper invokes to explain ReLU2's catastrophic dead-node collapse.","marker":"He et al. [2015]"},{"why":"Introduces RBF prototype-distance networks, the conceptual predecessor the paper compares OffsetL2 against.","marker":"Moody and Darken [1989]"}],"fun_headline_variants":["Second ReLU exposes neural nets' hidden distance bias","Why nets prefer distance: a negation layer proves it","Distance beats intensity in neural networks, OffsetL2 wins","Neural nets secretly measure distance, not signal strength"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument stands on the claim that the architecture constraints really force one representational type into the final layer: cross-entropy plus a non-negative activation forces intensity, and a negation layer converts distance into intensity—if the ReLU2 collapse is instead just the standard dying-ReLU optimization problem, the central distance-versus-intensity distinction loses its footing.","fun_headline_variants_meta":{"raw":{"variants":["Second ReLU exposes neural nets' hidden distance bias","Why nets prefer distance: a negation layer proves it","Distance beats intensity in neural networks, OffsetL2 wins","Neural nets secretly measure distance, not signal strength"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000117,"raw_usage":{"total_tokens":1034,"prompt_tokens":853,"completion_tokens":181,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":469,"completion_tokens_details":{"reasoning_tokens":117}},"tokens_in":469,"tokens_out":181,"duration_ms":2295,"temperature":1.0,"reasoning_tokens":117,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T13:19:34.459672+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a ReLU2-Neg network on MNIST, identify each class prototype by intersecting the first-layer hyperplanes, and measure the correlation between an input's distance to the class-c prototype and the class-c logit. The paper's claim predicts that smaller distances correspond to larger (less negative) or near-zero logits in a consistent ordering across all ten classes; if the logits instead vary with intensity-style separation and show no distance ordering, the distance-representation reading is falsified.","supporting_citations":[{"cited_title":"On the generalized distance in statistics","cited_arxiv_id":null,"evidence_quote":"Defines the Mahalanobis distance, the statistical metric the paper uses as the foundation for distance-based representations."},{"cited_title":"Fast learning in networks of locally-tuned processing units","cited_arxiv_id":null,"evidence_quote":"Introduces RBF prototype-distance networks, the conceptual predecessor the paper compares OffsetL2 against."}],"review_version":1}