REVIEW 3 major objections 5 minor 28 references
This paper claims that metric landscapes—maps of AUROC, AUPR, F1, and MCC over the sensitivity–specificity square—reveal how these metrics' values and meanings shift with the anomaly ratio, giving practitioners a reference for comparing sco
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-01 05:13 UTC pith:NCL4GC3K
load-bearing objection The F1/MCC landscapes and null baselines are genuinely useful, but the AUROC/AUPR landscapes rest on an unstated point-to-curve completion rule, so the paper's central guidance for two of its four metrics doesn't follow from the definitions. the 3 major comments →
An Insight on Evaluation Metrics Under the Imbalanced Case of Anomaly Detection
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The paper's central discovery is that the four standard metrics live on describable, imbalance-dependent surfaces over the (TPR, TNR) square. On these surfaces, AUROC and MCC remain centered around 0.5 independent of the anomaly ratio, while AUPR and F1 have baselines that rise with the proportion of positives—so a high AUPR or F1 value on a positive-dominant dataset may still be close to random performance. The paper further reports that AUROC and MCC retain near-identical structure across imbalance regimes, that AUPR requires no positive detections up to 0.75 at balance, and that in low-anomaly regimes the metrics decorrelate and reward different operating regions. The authors frame metric
What carries the argument
Metric landscapes—the maps of each metric value over the (TPR, TNR) unit square at a fixed imbalance ratio—are the central object. They are computed for AUROC, AUPR, F1, and MCC across ratios from 99:1 to 1:99 and built on a null model of random predictions (100 runs per ratio) to establish baselines. The landscape construction lets the authors compare regions of interest, measure minimal TPR/TNR needed to reach target scores, and run factor analysis on the landscapes to quantify how metrics correlate under low, medium, and high imbalance.
Load-bearing premise
The paper's landscape conclusions for AUROC and AUPR rest on the unstated assumption that these curve-based metrics can be evaluated at a single (TPR,TNR) operating point; if that single-point assignment does not match the standard integral definitions, the AUROC/AUPR landscape values are not the standard metrics.
What would settle it
Take a classifier with known TPR and TNR, construct the full ROC and PR curves (e.g., a single threshold point with linear interpolation), compute the textbook AUROC and AUPR integrals, and compare them to the values the paper's landscapes assign at those coordinates; disagreement would show the landscape's AUROC/AUPR surfaces are not the standard metrics.
If this is right
- If the landscapes are right, a given F1 or AUPR value should always be read against the anomaly ratio of the test set: values near 0.7 can be random in positive-dominant settings but meaningful in negative-dominant ones.
- AUROC and MCC remaining centered near 0.5 across imbalance means those two are the safer metrics for comparing models across datasets with different anomaly ratios, with AUROC and MCC behaving almost identically in high-positive regimes.
- The minimal-positive analysis implies that in balanced settings a model can reach AUPR 0.75 without detecting any anomalies, so AUPR gains below that threshold should not be credited to detection.
- In low-anomaly regimes the four metrics decorrelate, so a single metric cannot proxy for the others; evaluation should report several metrics.
- As anomaly ratio decreases, reaching a fixed F1 score requires an increasing share of correctly classified negatives, so F1 comparisons across datasets implicitly reward models that get the majority class right.
Where Pith is reading between the lines
- If the underlying single-point assignment for AUROC/AUPR is a piecewise-linear interpolation rather than the standard curve integrals, the paper's quantitative AUROC/AUPR landscape values (e.g., 0.75 baseline at balance) are about a different quantity; the qualitative preference structure may still hold.
- The landscape framework could be extended to threshold-dependent operating-point selection: practitioners could pick the (TPR,TNR) region their application values and read off the minimal imbalance-ratio-corrected score needed to trust a model.
- The same landscape construction could be applied to other metrics (e.g., balanced accuracy, G-mean, PR-AUC variants, cost-weighted scores) to check whether their imbalance behavior is similarly structured.
- A testable extension: on a real dataset, split predictions by anomaly ratio and build empirical landscapes; they should match the paper's synthetic surfaces if the framework captures actual metric behavior.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper analyzes four anomaly-detection evaluation metrics (AUROC, AUPR, F1, MCC) under varying class-imbalance ratios. Its main instrument is the 'metric landscape': a visualization of each metric as a function of (TPR, TNR) over the unit square, plotted at different anomaly ratios. The paper reports random-baseline statistics, landscape-difference plots, region-of-interest averages, and factor analyses. The central claim is that these landscapes provide an intuitive view of metric preferences and stability, and offer practical guidance for interpreting anomaly-detection results across datasets with differing imbalance ratios.
Significance. If the construction were well-defined, the proposed landscapes could serve as a useful visual reference for practitioners. The paper does provide a straightforward empirical characterization of F1 and MCC as pointwise functions of (TPR, TNR), and the random-baseline analysis is reproducible in principle. However, the central tool is fatally under-specified: AUROC and AUPR are defined as integrals over full ROC/PR curves, but the paper plots and interprets them at single (TPR, TNR) operating points without stating any rule for obtaining a scalar value from one point. The reported behaviors are inconsistent with the stated definitions, and no code or computation details are supplied. Consequently, the main results about AUROC and AUPR — including the minimal-positives analysis and the factor-analysis conclusions — do not pertain to the metrics as defined.
major comments (3)
- [§III-B, §V-A] The definitions of AUROC and AUPR in §III-B are curve integrals: AUROC = ∫ TPR(FPR^{-1}(x)) dx and AUPR = ∫ PPV(TPR^{-1}(x)) dx. A single operating point (TPR, TNR) does not determine either curve, so the value of AUROC or AUPR at that point is not identifiable. Yet Figures 2, 3, 4, 5, and 8 plot these metrics over the (TPR, TNR) square. The observed features, such as the AUROC baseline along TPR+TNR = 1 (§V-A), are not derivable from the integrals and appear to correspond to an ad hoc surrogate (e.g., (TPR+TNR)/2). The paper never states the interpolation or completion rule that maps a point to a curve. All AUROC/AUPR landscape conclusions are therefore about an unspecified quantity, not about the metrics defined in §III-B.
- [§V-C] The 'Minimal Positives Detection' analysis for AUROC and AUPR is built directly on the same undefined pointwise mapping. For instance, the claim that AUPR can reach 0.75 with zero TPR in the balanced case, or that AUROC requires no positive detection up to 0.5, is a property of the hidden interpolation rule, not of the standard AUROC/AUPR definitions. Without an explicit definition of the pointwise surrogate, these results are unreproducible and cannot be used as practical guidance. This is a load-bearing flaw, as the section's title and conclusions explicitly concern AUROC and AUPR.
- [§VI and Appendix B] The factor analysis (§VI) and score-coverage analysis (Appendix B) use landscape values as inputs. Since the AUROC/AUPR landscapes are not properly defined, the Pearson-correlation and Tucker-congruence results involving AUROC or AUPR are not meaningful as statements about the standard metrics. This undermines the cross-metric relationship claims, including the reported AUROC–MCC agreement and AUPR–AUROC decorrelation.
minor comments (5)
- [§V-D, R5] The text says AUROC and MCC 'remain stably centred around 0' in the 45–55% TPR/TNR region, but the plotted landscapes and region averages (Figure 13) indicate values around 0.5. This appears to be a typographical error and should be corrected.
- [Figure 7] The lower-triangle label 'Person's correlation' should be 'Pearson's correlation'.
- [§V-A] The phrase 'diagonal (TPR+TNR)/2 = 0.5' is confusing; the baseline line is TPR+TNR = 1, which is equivalent but should be written consistently.
- [Figure 1] The x-axis ordering runs from 99:1 to 1:99, while the text says 'as the proportion of positive samples increases (left side of the plot)'. Verify the direction and clarify the axis label.
- [Abstract/Introduction] There are minor grammatical issues, e.g., 'this work therefore study how metric values evolve' should be 'this work therefore studies...'.
Circularity Check
AUROC/AUPR landscapes are not the defined curve-integral metrics; the minimal-positive results reduce to an unstated pointwise construction.
specific steps
-
self definitional
[§III-B (AUROC/AUPR integral definitions); §V-A (landscapes); §V-C (minimal positives detection)]
"AUROC:=∫_0^1 TPR(FPR−1(x)) dx ... AUPR:=∫_0^1 PPV(TPR−1(x)) dx ... For AUROC, no positive sample needs to be detected up to a score of 0.5 ... For AUPR, no positive detection is required up to a score of 0.75 in a balanced dataset."
The paper defines AUROC and AUPR as integrals over entire ROC/PR curves, but then plots and interprets scalar values for each single (TPR,TNR) operating point without specifying how a point determines a full curve. A single operating point is compatible with infinitely many ROC/PR curves and hence with infinitely many AUROC/AUPR values. The 'minimal positives detection' claims in §V-C are therefore not consequences of the §III-B integral definitions; they are consequences of the unstated pointwise completion rule used to build the landscapes. The paper's own baseline observation, 'AUROC and MCC exhibit an approximately symmetric structure, with the baseline behaviour aligned along the diagonal (TPR+TNR)/2 = 0.5', reveals that the plotted AUROC object is a pointwise construction, not the in
full rationale
There is no fitting-to-data circularity and no load-bearing self-citation chain: the null analysis in §IV is descriptive and reproduces known random-baseline properties, F1 and MCC landscapes are genuine mathematical consequences of their confusion-matrix formulas, and the factor analysis is a post-hoc descriptive summary. The central circularity concern is limited to AUROC and AUPR. Because the paper never defines how a single (TPR,TNR) pair determines an entire ROC/PR curve, the landscape values for these two metrics are not the quantities defined in §III-B. Consequently, the practical guidance in §V-C and the coverage analysis in Appendix B inherit an unstated auxiliary construction. This is a partial, definitional circularity affecting the two threshold-independent metrics, while the F1/MCC portions retain independent content. A score of 4 reflects that the central visualization framework is compromised for AUROC/AUPR but is not a global reduction of the paper to its inputs.
Axiom & Free-Parameter Ledger
free parameters (1)
- factor-analysis settings
axioms (3)
- ad hoc to paper AUROC and AUPR are pointwise functions of a single (TPR,TNR) operating point
- domain assumption Imbalance ratios from Table I are representative of real anomaly-detection test distributions
- domain assumption Factor analysis with three factors and orthogonal rotation captures metric relationships
read the original abstract
Anomaly detection is inherently characterised by severe class imbalance, making the interpretation of evaluation metrics challenging. Although metrics such as AUROC, AUPR, F1-score, and MCC are widely used, their values convey different meanings depending on the anomaly ratio. In this work, we analyse the behaviour of those four common anomaly detection metrics under varying levels of imbalance. We focus on the study of metric landscapes, visualisations that relate metric values to true positive and true negative rates, providing an intuitive view of metric preferences and stability. Our analysis offers practical guidance for interpreting and comparing anomaly detection results across datasets with different imbalance ratios.
Figures
Reference graph
Works this paper leans on
-
[1]
Unsupervised anomaly detection in time-series: An exten- sive evaluation and analysis of state-of-the-art methods,
N. Mejri, L. Lopez-Fuentes, K. Roy, P. Chernakov, E. Ghorbel, and D. Aouada, “Unsupervised anomaly detection in time-series: An exten- sive evaluation and analysis of state-of-the-art methods,” Expert System with Application, 2024
2024
-
[2]
Common limitations of image processing metrics: A picture story,
A. Reinke and M. E. et al., “Common limitations of image processing metrics: A picture story,” Computing Research Repository, 2021
2021
-
[3]
Evaluating anomaly detection algorithms: A multi- metric analysis across variable class imbalances,
M. Hossain, M. S. Hossain, S. Kl ¨uttermann, and E. M ¨uller, “Evaluating anomaly detection algorithms: A multi- metric analysis across variable class imbalances,” in International Joint Conference on Neural Networks, 2024
2024
-
[4]
The advantages of the matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation,
D. Chicco and G. Jurman, “The advantages of the matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation,” BMC Genomics, 2020
2020
-
[5]
The impact of class imbalance in classification performance metrics based on the binary confusion matrix,
A. Luque, A. Carrasco, A. Mart ´ın, and A. de las Heras, “The impact of class imbalance in classification performance metrics based on the binary confusion matrix,” Pattern Recognition, 2019
2019
-
[6]
Performance of evaluation metrics for classification in imbalanced data,
A. de la Cruz Huayanay, J. L. Baz ´an, and C. M. Russo, “Performance of evaluation metrics for classification in imbalanced data,” Computational Statistics, 2025
2025
-
[7]
Halimu, A
C. Halimu, A. Kasem, and S. Newaz, “Empirical comparison of area under roc curve (AUC) and mathew correlation coefficient (MCC) for evaluating machine learning algorithms on imbalanced datasets for binary classification,” International Conference on Machine Learning and Soft Computing, 2019
2019
-
[8]
Comparison of evaluation metrics in classification applications with imbalanced datasets,
M. Fatourechi, R. K. Ward, S. G. Mason, J. Huggins, A. Schl ¨ogl, and G. E. Birch, “Comparison of evaluation metrics in classification applications with imbalanced datasets,” in International Conference on Machine Learning and Applications, 2008
2008
-
[9]
Choosing the right metrics: A study of performance measurement for binary classification in im- balanced and big data,
M. A. Walauskis and T. M. Khoshgoftaar, “Choosing the right metrics: A study of performance measurement for binary classification in im- balanced and big data,” Florida Artificial Intelligence Research Society, 2025
2025
-
[10]
On the classification of imbal- anced datasets,
C. V . KrishnaVeni and T. S. Rani, “On the classification of imbal- anced datasets,” International Journal of Computer Science Trends and Technology, 2011
2011
-
[11]
Balancing the scales: A comprehensive study on tackling class imbalance in binary classification,
M. Abdelhamid and A. Desai, “Balancing the scales: A comprehensive study on tackling class imbalance in binary classification,” Computing Research Repository, 2024
2024
-
[12]
On the classification of im- balanced datasets,
Arun Kumar M. N. and H. S. Sheshadri, “On the classification of im- balanced datasets,” in International Journal of Computer Applications, 2012
2012
-
[13]
A comparison of performance metrics with severely imbalanced network security big data,
T. Hasanin, T. M. Khoshgoftaar, and J. L. Leevy, “A comparison of performance metrics with severely imbalanced network security big data,” in International Conference on Information Reuse and Integration, 2019
2019
-
[14]
Evaluating classifier performance with highly imbalanced big data,
J. T. Hancock, T. M. Khoshgoftaar, and J. M. Johnson, “Evaluating classifier performance with highly imbalanced big data,” Journal of Big Data, 2023
2023
-
[15]
Evaluation metrics for anomaly detection algorithms in time-series,
G. Kov ´acs, G. Sebestyen, and A. Hangan, “Evaluation metrics for anomaly detection algorithms in time-series,” Acta Universitatis Sapientiae Informatica, 2019
2019
-
[16]
A study on performance metrics for anomaly detection based on industrial control system operation data,
G.-Y . Kim, S.-M. Lim, and I.-C. Euom, “A study on performance metrics for anomaly detection based on industrial control system operation data,” Electronics, 2022
2022
-
[17]
Navigating the metric maze: a taxonomy of evaluation metrics for anomaly detection in time series,
S. Sørbø and M. Ruocco, “Navigating the metric maze: a taxonomy of evaluation metrics for anomaly detection in time series,” Knowledge Discovery in Databases, 2024
2024
-
[18]
Current time series anomaly detection bench- marks are flawed and are creating the illusion of progress,
R. Wu and E. J. Keogh, “Current time series anomaly detection bench- marks are flawed and are creating the illusion of progress,” Transactions on Knowledge and Data Engineering, 2023
2023
-
[19]
W ADI: a water distribution testbed for research in the design of secure cyber physical systems,
C. M. Ahmed, V . R. Palleti, and A. P. Mathur, “W ADI: a water distribution testbed for research in the design of secure cyber physical systems,” in CySW ATER@CPSWeek, 2017
2017
-
[20]
Time series anomaly detection for cyber-physical systems via neural system identification and bayesian filtering,
C. Feng and P. Tian, “Time series anomaly detection for cyber-physical systems via neural system identification and bayesian filtering,” in Knowledge Discovery in Databases, 2021
2021
-
[21]
SWaT: a water treatment testbed for research and training on ICS security,
A. P. Mathur and N. O. Tippenhauer, “SWaT: a water treatment testbed for research and training on ICS security,” in CySW ATER@CPSWeek, 2016
2016
-
[22]
Practical approach to asyn- chronous multivariate time series anomaly detection and localization,
A. Abdulaal, Z. Liu, and T. Lancewicki, “Practical approach to asyn- chronous multivariate time series anomaly detection and localization,” in Knowledge Discovery in Databases, 2021
2021
-
[23]
Learning multiple layers of features from tiny images,
A. Krizhevsky, “Learning multiple layers of features from tiny images,” 2009
2009
-
[24]
Spot-the- difference self-supervised pre-training for anomaly detection and seg- mentation,
Y . Zou, J. Jeong, L. Pemula, D. Zhang, and O. Dabeer, “Spot-the- difference self-supervised pre-training for anomaly detection and seg- mentation,” in European Conference on Computer Vision, 2022
2022
-
[25]
MVTec AD - A comprehensive real-world dataset for unsupervised anomaly detection,
P. Bergmann, M. Fauser, D. Sattlegger, and C. Steger, “MVTec AD - A comprehensive real-world dataset for unsupervised anomaly detection,” in Conference on Computer Vision and Pattern Recognition, 2019
2019
-
[26]
Aster: Latent pseudo-anomaly genera- tion for unsupervised time-series anomaly detection,
R. Hermary, S. Hicsonmez, D. Pineau, A. E. R. Shabayek, and D. Aouada, “Aster: Latent pseudo-anomaly genera- tion for unsupervised time-series anomaly detection,” in International Conference on Pattern Recognition, 2026
2026
-
[27]
Removing geometric bias in one-class anomaly detection with adaptive feature perturbation,
R. Hermary, V . Gaudilli `ere, A. E. R. Shabayek, and D. Aouada, “Removing geometric bias in one-class anomaly detection with adaptive feature perturbation,” in Winter Conference on Applications of Computer Vision, 2025
2025
-
[28]
A survey on zero-shot image anomaly detection,
S. Hic ¸s ¨onmez, R. Hermary, D. Pineau, N. Mejri, A. Shabayek, and D. Aouada, “A survey on zero-shot image anomaly detection,” Preprint, 2026. An Insight on Evaluation Metrics Under the Imbalanced Case of Anomaly Detection –Supplementary– APPENDIXA DETAILS ONMETRICSELEMENTS Table II Summarises the different components of the metrics formulas discussed in...
2026
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.