{"id":"b3f43d58-edd3-49dd-beb4-1076ab8a24bb","arxiv_id":"2505.18032","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"L2-normalizing pre-logit features improves Mahalanobis-based out-of-distribution detection across 44 ImageNet models, reducing the average false positive rate by 7.6 percentage points.","lead":"This paper shows that normalizing a neural network's feature vectors before computing the Mahalanobis distance, a common out-of-distribution detector, consistently improves detection accuracy across 44 image models. The fix, called Mahalanobis++, is a simple projection that makes the features better fit the Gaussian assumption the detector relies on.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Normalization's benefit is conditional on feature-norm variance; 'consistent' claim overgeneralizes and should be conditional.","rationale":"The reader identified the same load-bearing premise: normalization must discard only confounding norm information while preserving discriminative direction. The paper's own exceptions and diagnostics show this premise is not universal. My concern sharpens this by pointing to evidence that norm can carry OOD-relevant signal (Figure 10) and that the robustness claim is overstated relative to Appendix Table 17. A direct correlation test would empirically establish whether the method's benefit is conditional on norm variance. If it is, the central claim of 'consistent' improvement should be qualified; if not, the current framing is acceptable. The reader's CONDITIONAL verdict already encompasses this, so no adjustment is needed.","tokens_in":51612,"tokens_out":6297,"duration_ms":48223,"concrete_test":"Compute, for each of the 44 models, the FPR improvement Δ = FPR(Maha) − FPR(Maha++) on the OpenOOD average (Table 4) and correlate it with a pre-registered norm-variation metric derived from the paper's own data: e.g., the variance-deviation score of Eq. 5 (Table 7) or the coefficient of variation of class-mean feature norms (Figure 8). Use Spearman rank correlation across the models with available norm stats. If ρ > 0.6 with p < 0.05, the benefit is strongly conditional and the abstract should be revised to state that normalization helps models with high norm variance; if ρ is near zero, the benefit is more universal and the current emphasis is acceptable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that L2 normalization 'significantly and consistently' improves Mahalanobis-based OOD detection rests on the premise that feature norm is purely a confounder. The paper's own diagnostics contradict this premise. Figure 10 shows that for several models (e.g., ViT-B16-orig, ViT-B16-miil, ViT-B16-clip), OOD samples (NINCO) have larger feature norms than ID samples, so norm carries class-discriminative signal. The augreg ViTs, which already have low norm variance (Table 7, Figure 8), show degraded FPR under normalization (ViT-B16-In21k-augreg: 25.7→28.3; ViT-L16-In21k-augreg: 21.0→23.9; ResNet50: 49.5→52.0 in Table 4). Thus normalization is beneficial only when norm variance is pathologically high; it is not a universally applicable fix. The abstract and Section 4 claim 'all considered cases' and 'consistently' without stating this conditionality. Moreover, the robustness claim in Table 5 ('remedies brittleness') is contradicted by Appendix Table 17, where several models still fail many unit tests (e.g., ConvNeXt-B: 15, DeiT3-B16: 15, DeiT3-L16: 8). The method's advantage is real but conditional; the paper should report the boundary of applicability.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Mahalanobis++, a post-hoc OOD detection method that applies l2-normalization to pre-logit features before estimating and using the Mahalanobis distance. The authors argue that feature norm variations violate the Gaussian shared-covariance assumption of the Mahalanobis score, and that projecting features onto the unit sphere reduces this violation and decouples the score from feature norm. They evaluate the method on 44 ImageNet checkpoints and several CIFAR models, reporting an average FPR improvement of 7.6 points over standard Mahalanobis on OpenOOD datasets, 10.9 points on the cleaned NINCO subset, and state-of-the-art average performance relative to a broad set of baselines. The paper also claims that normalization remedies the brittleness of Mahalanobis-based detectors on noise unit tests.","tokens_in":51895,"tokens_out":7528,"duration_ms":61116,"significance":"If the empirical claims are properly qualified, this is a practically valuable contribution: the method is simple, adds no fitted parameters, and is applicable to any pretrained model. The evidence base is unusually strong for a post-hoc OOD paper—44 diverse checkpoints, five OOD datasets, multiple architectures, and three complementary diagnostic analyses (norm distributions, QQ plots, variance-alignment scores). The hypothesis was formed on one model and validated on held-out checkpoints, and code is provided, which increases confidence and reproducibility. The main weakness is that the abstract and several central statements overgeneralize: the paper's own results show three models where normalization degrades FPR, and the full unit-test table shows many models still fail a substantial number of noise tests. These overclaims, if corrected, do not undermine the core averaged improvement but they do require a revised, conditional formulation of the method's benefits.","major_comments":[{"comment":"The claim that l2-normalization 'improves the conventional Mahalanobis distance-based approaches significantly and consistently' is contradicted by the paper's own Table 4: three models (ViT-B16-In21k-augreg, ViT-L16-In21k-augreg, ResNet50) show worse FPR after normalization (25.7→28.3, 21.0→23.9, 49.5→52.0). Appendix D attributes these exceptions to already well-behaved feature norms, which implies the benefit is conditional on the degree of norm variance. The abstract, Section 4, and conclusion should state this boundary explicitly rather than claiming consistency.","section":"Abstract, §4, Table 4"},{"comment":"The main text states that normalization 'remedies the brittleness' of Mahalanobis-based detectors and Table 5 highlights three models with perfect 0/17 unit-test outcomes, but the full results in Table 17 show that many models still fail a substantial number of unit tests after normalization (e.g., ConvNeXt-B: 15, DeiT3-B16: 15, DeiT3-L16: 8, DeiT3-S16-In21k: 11, Mixer-B16-In21k: 10), and for ResNet101 and ResNet50 the failure count increases from 0 to 1. The robustness claim should be rephrased as 'reduces the number of failures for most models' and the full distribution of failure counts should be summarized in the main text.","section":"§5, Table 5, Appendix Table 17"},{"comment":"The paper treats feature norm as a pure confounder, but Figure 10 shows that for several models (e.g., ViT-B16-orig, ViT-B16-miil, ViT-B16-clip) natural OOD samples from NINCO have larger feature norms than ID samples, meaning the norm carries class-discriminative signal that l2-normalization discards. The diagnostic evidence for the confounder claim is built primarily on SwinV2 and models with small OOD norms; it does not establish that discarding the norm is always safe. The paper should explicitly state that normalization is beneficial when the norm-OOD-score correlation is confounded (as in Figure 5) and should report conditions under which it helps, e.g., using the variance-alignment metric of Table 7 or the norm overlap of Figure 10 as a predictor.","section":"§3.3, Figure 10, §4"}],"minor_comments":[{"comment":"There is a typo: 'safety-critial' should be 'safety-critical'.","section":"Abstract"},{"comment":"The reference 'Lee et al. (2018a)' for the Gaussian assumption should likely be 'Lee et al. (2018b)', since the cited 2018a paper is about confidence calibration rather than the Mahalanobis framework.","section":"§3.1"},{"comment":"The subsection 'Improved normality' claims that normalization makes features 'more aligned with a normal distribution,' but after l2-normalization the features lie on the unit sphere, which cannot be multivariate Gaussian. The QQ-plot evidence on centered, standardized coordinates is an indirect diagnostic; the text should not literally assert that normalized features are better modeled as a shared-covariance Gaussian.","section":"§4"},{"comment":"The criteria for selecting the three reported models (ConvNeXtV2, SwinV2, ViT-CLIP) should be stated; without this, the selection appears arbitrary and potentially cherry-picked relative to the full results in Table 17.","section":"Table 5"},{"comment":"The sentence 'In 30/44 cases, the best performing method is Mahalanobis++ (in 6/44 cases it is relative Mahalanobis++)' should clarify what happens in the remaining 8 cases, e.g., whether they are ties or another method.","section":"§5, ImageNet"},{"comment":"The vertical line labeled 'True sample norm' is unclear; the caption should describe that this marks the unmodified feature norm of the OOD samples.","section":"Figure 6"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a solid empirical contribution with an unusually broad evaluation and reproducible code. The main issue is not the validity of the averaged improvement but the overgeneralized wording in the abstract and robustness claims. I recommend major revision so that the authors can qualify the consistency claim and the unit-test conclusion, and state the conditions under which normalization helps or hurts."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Max, the short version: this paper has a genuinely useful result and an honest evaluation, but the abstract and a few claims overstate the universality. The method is exactly what the title says—L2-normalize pre-logit features before estimating and applying the Mahalanobis distance. On 44 ImageNet checkpoints it cuts average FPR by 7.6 points and beats ViM by about 7 points; on the cleaned NINCO subset the gain is 10.9 points. That is a real post-hoc upgrade for a widely used detector, with no hyperparameters and no retraining.\n\nThe diagnostic work is the more interesting part. The paper shows that feature-norm variance, both across and within classes, correlates with the Mahalanobis score and flags it as a violation of the shared-Gaussian assumption. The QQ-plots, the variance-alignment metric (Eq. 5), and the norm-scaling experiment in the appendix nail the mechanism. The derivation of Lemma 3.1 and the Eq. 5 moment calculation are in the appendix and check out. This is reproducible: code is available, the evaluation covers 44 models with diverse architectures and pretrainings, and the appendix has the full per-model tables.\n\nNow the soft spots. 'Consistently' in the abstract is not what the data say. Three models degrade (two augreg ViTs and ResNet50), and the paper's own Table 12 shows that Maha++ is not the best in 2 of 44 cases against cosine-based methods, contradicting the 'all considered cases' claim in Section 4. The unit-test robustness story is also oversold: Table 5 says Maha++ fails 0/17 unit tests for three models, but Table 17 shows ConvNeXt-B still fails 15, DeiT3-B16 15, DeiT3-L16 8. The paper does acknowledge the augreg exception and discuss the three outliers, so the data are there; the problem is the main-text framing. The benefit is conditional—normalization helps when norm variance is pathologically high, and can hurt when the features are already roughly Gaussian. That boundary should be in the abstract.\n\nAll in all, the central claim holds for the great majority of models and the diagnosis is well supported. It deserves a serious referee; the revision just needs to align the claims with the full tables. I'd take it for our reading group.","headline":"Simple post-hoc fix that works on 41 of 44 models, but 'consistent' and 'all considered cases' are overstatements that need aligning with the paper's own tables.","tokens_in":52407,"tokens_out":2887,"would_cite":true,"duration_ms":24759,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"L2-normalizing pre-logit features before the Mahalanobis score makes OOD detection consistently stronger across 44 ImageNet models.","keywords":["out-of-distribution detection","Mahalanobis distance","feature normalization","L2 normalization","pre-logit features","Gaussian assumption","ImageNet evaluation","post-hoc methods"],"falsifier":"On a model whose feature norms still correlate with the Mahalanobis++ score after normalization, compute the rank correlation between original feature norm and the normalized OOD score over in- and out-of-distribution test sets; the paper's mechanism predicts near-zero correlation, so a large residual correlation would falsify the claim that normalization decouples the score from feature scale.","tokens_in":51399,"feed_emoji":"🛡️","tokens_out":7274,"duration_ms":58872,"temperature":0.7,"pith_summary":"Post-hoc out-of-distribution (OOD) detection has to work on already-trained classifiers. This paper identifies why the Mahalanobis distance, one of the most effective post-hoc detectors, is inconsistent across models: the pre-logit features it relies on violate the assumed Gaussian model, with feature norms that vary far more across and within classes than a shared-covariance Gaussian would produce, and small-norm OOD samples being systematically scored as in-distribution. The proposed fix, Mahalanobis++, normalizes each pre-logit feature vector to unit length before estimating class means, the shared covariance matrix, and the final score. Across 44 ImageNet models the normalization improves average false positive rate by 7.6 points over the unnormalized Mahalanobis baseline, outperforms the previous best post-hoc method by about 7 points on average, and removes the fragility on simple noise distributions for several models. If the claim holds, practitioners gain a hyperparameter-free modification that works across architectures and pretraining schemes without retraining.","feed_headline":"Normalizing features fixes Mahalanobis OOD detection","feed_subtitle":"L2-normalization before the Mahalanobis score cuts average false positive rate by 7.6 points across 44 ImageNet models.","key_machinery":"The machinery is the pre-logit feature vector $\\phi(x)$, the Mahalanobis distance $d_{\\mathrm{Maha}}(x,\\hat{\\mu}_c)=(\\phi(x)-\\hat{\\mu}_c)^T\\hat{\\Sigma}^{-1}(\\phi(x)-\\hat{\\mu}_c)$ to the class mean $\\hat{\\mu}_c$, and the replacement of $\\phi(x)$ by $\\hat{\\phi}(x)=\\phi(x)/\\|\\phi(x)\\|_2$ in every step: class-mean estimation, shared-covariance estimation, and scoring. This projects all features onto the unit sphere and discards scale information. The paper motivates the replacement with a concentration lemma showing that truly Gaussian high-dimensional features have concentrated norms, with QQ plots and a variance-alignment score showing that real features are heavy-tailed and class-dependent, and with a documented correlation between feature norm and the unnormalized Mahalanobis score that normalization breaks.","core_discovery":"The paper's central claim is that L2-normalizing pre-logit features before computing the Mahalanobis distance makes a brittle OOD detector consistently strong, because the normalization makes the empirical feature distribution better satisfy the class-conditional Gaussian model with shared covariance that the Mahalanobis score implicitly assumes. Evidence is the 44-model ImageNet study and CIFAR experiments: Mahalanobis++ improves FPR over unnormalized Mahalanobis in 41 of 44 models, with average gains of 7.6 FPR points on the five-dataset OpenOOD suite and 10.9 points on the cleaned NINCO dataset. The only models that do not benefit are those whose feature norms are already tightly concentrated—notably two augreg-trained ViTs—which is consistent with the explanation rather than a refutation of it.","pith_inferences":["The pre-training feature-norm diagnostics could serve as an OOD-data-free early check: a model whose class norms are spread or correlated with the Mahalanobis score is a candidate for normalization, while a model like the augreg ViT is not.","Any distance- or norm-based OOD score may inherit the same scale confound, so L2 normalization is likely to help other post-hoc detectors, not only the Mahalanobis family; the paper compares against but does not make this claim.","Because well-behaved features in this paper come from a specific augmentation-and-regularization recipe whose mechanism is left unexplained, the results hint that training schemes could deliberately induce norm-regulated features and make post-hoc OOD detection uniformly reliable."],"forward_implications":["Mahalanobis++ outperforms the conventional Mahalanobis distance in 41 of 44 ImageNet models and beats the strongest previous post-hoc baseline, ViM, by about 7 average FPR points.","Because it is post-hoc and hyperparameter-free, the method applies directly to any frozen pretrained classifier, including models whose pretraining data or procedure is not publicly available.","Normalization repairs the known brittleness of Mahalanobis-based detectors on far-OOD noise distributions: for several representative models the number of failed unit tests drops from 10-14 to 0.","The gain is largest on NINCO, where contamination with in-distribution images is removed, with an average improvement of 10.9 FPR points and a new best FPR of 18.4%.","On CIFAR100 the recipe improves Mahalanobis in every tested model, but by smaller margins, consistent with weaker norm pathology at smaller scale."],"supporting_citations":[{"why":"Proposes the Mahalanobis-distance OOD score on pre-logit features that Mahalanobis++ modifies and evaluates against.","marker":"(Lee et al., 2018b)"},{"why":"Introduces the relative Mahalanobis distance, the closest competitor whose normalized version relative Mahalanobis++ is also reported.","marker":"(Ren et al., 2021)"},{"why":"Supplies the NINCO benchmark and the noise unit tests that expose the brittleness Mahalanobis++ fixes.","marker":"(Bitterwolf et al., 2023)"},{"why":"Defines the augreg training scheme; its ViT checkpoints are the main models where normalization gives no improvement because features are already well-behaved.","marker":"(Steiner et al., 2022)"},{"why":"Provides the deep-nearest-neighbor baseline operating in normalized feature space that Mahalanobis++ consistently outperforms.","marker":"(Sun et al., 2022)"},{"why":"Proposes ViM, the strongest non-Mahalanobis baseline that Mahalanobis++ beats by about 7 average FPR points.","marker":"(Wang et al., 2022)"},{"why":"Provides the SwinV2 model family used for the main analysis of feature-norm pathologies and the correlation with the Mahalanobis score.","marker":"(Liu et al., 2022)"},{"why":"Defines the OpenOOD benchmark protocol and dataset suite used for the 44-model evaluation.","marker":"(Yang et al., 2022)"}],"fun_headline_variants":["L2 normalization boosts Mahalanobis OOD detection","Normalize features to fix Mahalanobis OOD detection","Mahalanobis++: L2 norm makes OOD detection reliable","Feature normalization improves Mahalanobis OOD scores","L2 norm before Mahalanobis improves OOD detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Normalizing away each sample's feature length removes only harmless scale information while preserving the directional pattern that separates classes; if a model carries its OOD signal in the feature norm itself, this fix can backfire.","fun_headline_variants_meta":{"raw":{"variants":["L2 normalization boosts Mahalanobis OOD detection","Normalize features to fix Mahalanobis OOD detection","Mahalanobis++: L2 norm makes OOD detection reliable","Feature normalization improves Mahalanobis OOD scores","L2 norm before Mahalanobis improves OOD detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000169,"raw_usage":{"total_tokens":1223,"prompt_tokens":866,"completion_tokens":357,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":482,"completion_tokens_details":{"reasoning_tokens":272}},"tokens_in":482,"tokens_out":357,"duration_ms":3194,"temperature":1.0,"reasoning_tokens":272,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:36:12.474703+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a model whose feature norms still correlate with the Mahalanobis++ score after normalization, compute the rank correlation between original feature norm and the normalized OOD score over in- and out-of-distribution test sets; the paper's mechanism predicts near-zero correlation, so a large residual correlation would falsify the claim that normalization decouples the score from feature scale.","supporting_citations":[{"cited_title":"In or out? fixing imagenet out-of-distribution detection evaluation","cited_arxiv_id":null,"evidence_quote":"Supplies the NINCO benchmark and the noise unit tests that expose the brittleness Mahalanobis++ fixes."},{"cited_title":"P., Kolesnikov, A., Zhai, X., Wightman, R., Uszkoreit, J., and Beyer, L","cited_arxiv_id":null,"evidence_quote":"Defines the augreg training scheme; its ViT checkpoints are the main models where normalization gives no improvement because features are already well-behaved."},{"cited_title":"Out-of-distribution detection with deep nearest neighbors","cited_arxiv_id":null,"evidence_quote":"Provides the deep-nearest-neighbor baseline operating in normalized feature space that Mahalanobis++ consistently outperforms."},{"cited_title":"Vim: Out-of-distribution with virtual-logit matching","cited_arxiv_id":null,"evidence_quote":"Proposes ViM, the strongest non-Mahalanobis baseline that Mahalanobis++ beats by about 7 average FPR points."},{"cited_title":"OpenOOD: Benchmarking Generalized Out-of-Distribution Detection","cited_arxiv_id":"2210.07242","evidence_quote":"Defines the OpenOOD benchmark protocol and dataset suite used for the 44-model evaluation."}],"review_version":1}