The reviewed record of science sign in
Pith

arxiv: 2606.02601 · v1 · pith:T2CDFD4U · submitted 2026-05-23 · cs.LG

Testing the Test: Score-Direction Instability in Class-Split Anomaly Detection

Alejandro Ascarate , Leo Lebrat , Rodrigo Santa Cruz , Clinton Fookes , Olivier Salvado This is my paper

Reviewed by Pith2026-06-30 14:09 UTCgrok-4.3pith:T2CDFD4Uopen to challenge →

classification cs.LG
keywords anomaly detectionclass-split evaluationout-of-distribution detectionrepresentation space overlapscore instabilityneighborhood leakageFashion-MNISTCIFAR-10
0
0 comments X

The pith

Class-split anomaly detection tests can produce inverted or random scores when the held-out class overlaps normals in representation space.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that within-dataset class-split evaluation, a common proxy for unconditional out-of-distribution anomaly detection, becomes ill-posed when the anomaly class overlaps the normal mixture in representation space. In this overlap regime, anomaly scores collapse toward chance performance or invert, and which score direction is preferred can depend on the specific unknown anomaly class. The authors introduce neighborhood class leakage, a training-free diagnostic computed from representation-space neighborhoods, and demonstrate that it predicts this instability across Fashion-MNIST, CIFAR-10, and Imagenette in both pixel and VAE latent spaces. If correct, this means many existing class-split benchmarks cannot be treated as unconditional evidence of anomaly-detection ability and must instead be viewed as geometry-dependent stress tests.

Core claim

When the held-out anomaly class overlaps the normal mixture in representation space, anomaly scores may collapse toward chance or invert, and the preferred score direction depends on the unknown anomaly class. Neighborhood class leakage, a simple training-free measure of how much anomaly points mix into normal neighborhoods in representation space, predicts this score-direction instability across multiple datasets and representation spaces.

What carries the argument

neighborhood class leakage: a training-free diagnostic that measures the fraction of anomaly-class points appearing in local neighborhoods of normal points in representation space, used to forecast when anomaly scores will become unstable or directionally dependent on the held-out class.

If this is right

  • Reported performance numbers from class-split anomaly detection experiments on CIFAR-10 and similar datasets may reflect representation geometry rather than intrinsic detection ability.
  • The same anomaly detection model can appear effective or ineffective depending on which class is treated as anomalous, undermining comparisons across papers.
  • Evaluation protocols should incorporate checks for representation overlap before interpreting scores as evidence of out-of-distribution capability.
  • True unconditional anomaly detection requires test conditions where anomaly classes are guaranteed to be separated in the chosen representation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Practitioners deploying anomaly detectors on real data with unknown anomaly types may need to monitor representation neighborhoods at inference time to decide whether to trust a given score direction.
  • This instability suggests that future benchmarks could benefit from explicitly constructing or selecting held-out classes that control for measured overlap rather than using fixed class splits.
  • The diagnostic could be extended to other unsupervised settings where performance depends on unknown test distributions, such as certain forms of clustering or density estimation.

Load-bearing premise

Neighborhood class leakage computed from representation-space neighborhoods reliably predicts score-direction instability without requiring labeled anomaly data or model retraining.

What would settle it

Run the class-split protocol on a new dataset where neighborhood class leakage is high for several held-out classes, then check whether anomaly scores remain stable in direction and above chance regardless of which class is held out.

Figures

Figures reproduced from arXiv: 2606.02601 by Alejandro Ascarate, Clinton Fookes, Leo Lebrat, Olivier Salvado, Rodrigo Santa Cruz.

Figure 1
Figure 1. Figure 1: Per-class AUROC on CIFAR-10 in pixel space under the 9-vs.-1 class-split protocol. The dashed line marks chance performance. Different held-out anomaly classes induce both above-chance and inverted rankings, illustrating score-direction instability. Imagenette (Howard, 2019; Russakovsky et al., 2015). For each dataset, we run the K − 1 vs. 1 class-split protocol, sweeping the held-out anomalous class c. We… view at source ↗
read the original abstract

Within-dataset class-split evaluation is widely used as a proxy for fully unconditional out-of-distribution anomaly detection. We show that this protocol can become ill-posed when the held-out anomaly class overlaps the normal mixture in representation space. In this regime, anomaly scores may collapse toward chance or even invert, and the preferred score direction can depend on the unknown anomaly class. We introduce a simple training-free diagnostic, neighborhood class leakage, and show that it predicts score-direction instability across Fashion-MNIST, CIFAR-10, and Imagenette, in both pixel and VAE latent spaces. Our results suggest that class-split AD benchmarks should be treated as geometry-dependent stress tests rather than unconditional evidence of anomaly-detection ability.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

0 major / 2 minor

Summary. The paper claims that within-dataset class-split evaluation for anomaly detection becomes ill-posed when the held-out anomaly class overlaps the normal mixture in representation space, causing anomaly scores to collapse toward chance or invert with the preferred direction depending on the unknown class. It introduces neighborhood class leakage as a training-free diagnostic computed from representation-space neighborhoods that predicts this score-direction instability, with empirical support across Fashion-MNIST, CIFAR-10, and Imagenette in both pixel and VAE latent spaces.

Significance. If the result holds, the work is significant for anomaly detection research because it demonstrates that standard class-split benchmarks may not provide unconditional evidence of detection ability but instead act as geometry-dependent stress tests. The empirical consistency across three datasets and two representation spaces, combined with the training-free and label-free nature of the diagnostic, provides a practical contribution that could improve benchmark interpretation without requiring model retraining or labeled anomalies.

minor comments (2)
  1. The abstract states that the diagnostic predicts instability across the listed datasets and spaces, but the manuscript would benefit from explicit reporting of error bars, statistical tests, or exclusion criteria for the empirical patterns to allow full verification of robustness.
  2. Clarify the precise definition and computation of neighborhood class leakage (e.g., neighborhood size, distance metric, and aggregation) in the methods to ensure the diagnostic can be reproduced independently.

Simulated Author's Rebuttal

0 responses · 0 unresolved

We thank the referee for the positive summary, significance assessment, and recommendation of minor revision. The report contains no enumerated major comments to address point by point.

Circularity Check

0 steps flagged

No significant circularity detected

full rationale

The paper presents an empirical finding that class-split anomaly detection can become ill-posed under representation-space overlap, supported by experiments on multiple datasets (Fashion-MNIST, CIFAR-10, Imagenette) in pixel and VAE spaces. The introduced diagnostic (neighborhood class leakage) is defined directly from representation-space neighborhoods on held-out classes without reference to the anomaly scores under test, without parameter fitting to those scores, and without load-bearing self-citations or uniqueness theorems. No equations or derivations reduce the reported predictions or instability observations to the inputs by construction; the central claim remains an independent empirical result rather than a tautology or renamed fit.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 1 invented entities

The central claim rests on the empirical observation that overlap in representation space produces instability; no free parameters are fitted to produce the main result, the diagnostic is a new derived metric rather than an invented physical entity, and background assumptions are standard supervised learning geometry.

axioms (1)
  • domain assumption Class-split evaluation within a single dataset serves as a reasonable proxy for unconditional out-of-distribution detection when classes are sufficiently separated.
    This is the protocol the paper is stress-testing; the abstract treats it as the default assumption being challenged.
invented entities (1)
  • neighborhood class leakage no independent evidence
    purpose: Training-free diagnostic that predicts score-direction instability from representation-space neighborhood overlap.
    New metric introduced in the paper; no independent evidence outside the reported experiments is provided.

pith-pipeline@v0.9.1-grok · 5658 in / 1389 out tokens · 45135 ms · 2026-06-30T14:09:26.737705+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

12 extracted references · 2 canonical work pages · 2 internal anchors

  1. [1]

    Deep Learning for Anomaly Detection: A Survey

    Deep Learning for Anomaly Detection: A Survey , author =. arXiv preprint arXiv:1901.03407 , year =

    work page internal anchor Pith review Pith/arXiv arXiv 1901

  2. [2]

    ACM Computing Surveys , volume =

    Deep Learning for Anomaly Detection: A Review , author =. ACM Computing Surveys , volume =. 2021 , doi =

    2021

  3. [3]

    Learning Multiple Layers of Features from Tiny Images , author =

  4. [4]

    NIPS Workshop on Deep Learning and Unsupervised Feature Learning , year =

    Reading Digits in Natural Images with Unsupervised Feature Learning , author =. NIPS Workshop on Deep Learning and Unsupervised Feature Learning , year =

  5. [5]

    Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms

    Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms , author =. arXiv preprint arXiv:1708.07747 , year =. doi:10.48550/arXiv.1708.07747 , url =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.1708.07747

  6. [6]

    2019 , howpublished =

    Imagenette , author =. 2019 , howpublished =

    2019

  7. [7]

    International Journal of Computer Vision , volume =

    ImageNet Large Scale Visual Recognition Challenge , author =. International Journal of Computer Vision , volume =. 2015 , doi =

    2015

  8. [8]

    International Conference on Learning Representations (ICLR) , year =

    Auto-Encoding Variational Bayes , author =. International Conference on Learning Representations (ICLR) , year =

  9. [9]

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages =

    Deep Residual Learning for Image Recognition , author =. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages =. 2016 , doi =

    2016

  10. [10]

    2008 Eighth IEEE International Conference on Data Mining , pages =

    Isolation Forest , author =. 2008 Eighth IEEE International Conference on Data Mining , pages =. 2008 , doi =

    2008

  11. [11]

    and Kriegel, Hans-Peter and Ng, Raymond T

    Breunig, Markus M. and Kriegel, Hans-Peter and Ng, Raymond T. and Sander, J. Proceedings of the 2000 ACM SIGMOD International Conference on Management of Data , pages =. 2000 , doi =

    2000

  12. [12]

    An Introduction to

    Fawcett, Tom , journal =. An Introduction to. 2006 , doi =

    2006