REVIEW 4 major objections 4 minor 42 references
Multi-class Image Anomaly Detection for Practical Applications: Requirements and Robust Solutions
T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Hierarchical memory banks let a single multi-class anomaly detector keep its accuracy when class labels are absent, while cutting memory-bank construction cost by more than an order of magnitude on large datasets.
desk verdict Useful taxonomy, but the label-robustness claim is undermined by suspiciously identical numbers across known/unknown conditions and an untested threshold mechanism. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the hierarchical memory bank. Semantic keys $c_k = \frac{1}{N_k}\sum_i e_i$ are centroids of cluster embeddings, each cluster's bank $M^*_k$ is a coreset of local patch features chosen by k-center greedy selection from patch features $\mathcal{P}(\phi_j(x_i))$, and inference assigns an input to cluster $k^* = \arg\min_k d(e, c_k)$ before scoring patches by nearest-neighbour distance within $M^*_{k^*}$, upsampled to an anomaly map. This structure replaces one global PatchCore bank with $K$ small banks, which is what lets the model define class-wise thresholds without labels and reduces complexity from $P^2(d+3r)$ to $\sum_i P_i^2(d+3r)+\mathcal{O}(N^2)$, where the FINCH term is negligible when the number of images is far smaller than the number of patches.
What would settle it
Re-run the unknown-class evaluation with thresholds chosen from normal training data only, for instance the per-cluster 95th percentile of normal image scores, instead of the oracle threshold used in Table 4; if HierCore's unknown-evaluation F1 drops noticeably, the claim that class labels are unnecessary at evaluation is not supported. Comparing those results with the cluster-specific thresholds described in Section 3.3 would also test whether the robustness mechanism works as intended.
Extended reading notes
Core claim
The central claim is that a single memory-bank model can satisfy both requirements the paper defines: it should be trainable with or without class labels, and it should stay strong under both known-class and unknown-class evaluation. HierCore achieves this by estimating pseudo-classes from a pre-trained encoder's final-layer embeddings using FINCH clustering selected by Silhouette score, building separate coreset memory banks of local patch features for each pseudo-class, and matching each test image to the nearest semantic cluster key before scoring it against that cluster's bank. Because each semantic cluster gets its own decision threshold, the model no longer depends on label information to separate normal from anomalous patterns. Under unknown-class evaluation with a single global threshold, the paper reports that HierCore's Diff Ratio, the ratio of unknown-evaluation F1 to known-evaluation F1, reaches 100% or 100.1% across all four datasets at both image and pixel levels, while every baseline drops substantially.
Load-bearing premise
The paper's label-free robustness result assumes that a single threshold chosen using ground-truth anomalous samples behaves the same as a threshold a user could set without knowing which images are defective; if that assumption fails, the 100% Diff Ratio is not a valid measure of real-world performance, and the paper never evaluates the cluster-specific thresholds it describes in Section 3.3.
Editorial extensions
If this is right
- A single memory-bank model can match the accuracy of per-class PatchCore in multi-class settings while also working without class labels, so scalability need not be bought with lower detection accuracy.
- Unknown-class evaluation changes model rankings: PatchCore and reconstruction-based multi-class models lose between 3% and 19% of their known-class F1 when a single global threshold replaces class-specific ones, while HierCore loses about 0%.
- HierCore cuts memory-bank construction time by about 13x on MVTecAD and more than 12x on VisA compared with PatchCore, and speeds up inference by about 3.5x with more stable frames-per-second performance.
- Class labels during training are not required: semantically clustered memory banks are at least as stable as label-based ones, and on MPDD semantic clusters split a single class label into multiple groups, suggesting labels can misalign with the structure useful for detection.
- Evaluation protocols that assume known classes at test time overstate practical robustness, so reporting the unknown-class Diff Ratio alongside the usual metrics would give a more honest picture of deployability.
Reading between the lines
- Beyond the paper: the unknown-evaluation protocol in Table 4 sets the global threshold using ground-truth anomalous samples; a stricter deployment test would set per-cluster thresholds from normal training scores only, which would show whether the 100% Diff Ratio survives truly label-free use.
- Beyond the paper: the hierarchical-bank design is a plug-in change for any memory-bank detector, so the same semantic-clustering step could be tested on medical or video anomaly benchmarks where class labels are noisy or absent.
- Beyond the paper: the paper describes but never evaluates the cluster-specific thresholds from Section 3.3; comparing them against the single global threshold would isolate whether the label-free gain comes from per-cluster thresholds or from the semantic grouping itself.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper formalizes two requirements for multi-class unsupervised image anomaly detection (MC-UIAD): the model should be trainable with or without class labels, and its evaluation performance should be stable whether or not class labels are available at inference. It proposes HierCore, a memory-bank model that first performs semantic clustering of normal images, builds per-cluster coreset memory banks, and at inference matches each test image to the nearest cluster and computes patch-level nearest-neighbor anomaly scores. The authors evaluate existing MC-UIAD methods and HierCore in four training/evaluation label-availability scenarios on MVTec AD, VisA, MPDD, and BTAD, reporting image- and pixel-level metrics including mF1-max and a composite mAD score. They also report memory-bank construction and inference speedups relative to PatchCore. The central claim is that HierCore consistently meets both requirements and maintains strong, stable performance across all settings.
Significance. If the central claim is established, the paper would make a genuinely useful practical contribution: a single memory-bank model that works across all four scenarios, including unknown-class training and evaluation, without requiring class labels for threshold selection. The scenario taxonomy itself (known/unknown labels in training/evaluation) is a useful organizational contribution, and the paper includes a broad benchmark of existing methods under this taxonomy. The efficiency analysis over PatchCore is concrete and well quantified. However, the load-bearing evidence for label-free robustness is currently not credible: the unknown-class evaluation uses a single optimal threshold chosen from ground-truth labels, and the cluster-specific threshold mechanism described in Section 3.3 is never actually exercised in the experiments. The paper has the right ingredients and a reasonable method, but its main empirical demonstration of Requirement 2 needs to be redone.
major comments (4)
- [Section 4.2, threshold definition] The evaluation protocol defines the threshold as the F1-maximizing value on the precision--recall curve, which requires ground-truth normal/anomaly labels for the test set. In the unknown-class evaluation, class labels are assumed absent, but anomaly labels are still used to select the threshold. This is not a label-free evaluation protocol, and it does not demonstrate that HierCore can set decision thresholds without annotation. The paper should either report threshold-independent metrics (e.g., AUROC) for the unknown condition or propose and validate a practical threshold estimator that uses only normal training data.
- [Section 3.3 vs. Table 4] Section 3.3 states that cluster-specific thresholds are used to detect anomalies independent of class labels, but Table 4's unknown-class evaluation applies a single optimal threshold across all classes. The proposed mechanism is therefore not tested. Please report HierCore results using the cluster-specific thresholds described in Section 3.3, and clarify how these thresholds are computed without access to ground-truth anomaly labels.
- [Table 4 and Appendix B] For HierCore, the known and unknown F1 scores are identical in nearly every dataset and in nearly every class row (e.g., MVTec image-level 97.9/97.9, VisA 92.2/92.2, BTAD 92.8/92.8, and the vast majority of class entries in Tables B.11--B.14). Under genuinely different thresholding protocols (per-class optimal vs. single global), exact equality is implausible given the inter-class threshold variation documented in Figure 2. This strongly suggests either the known-class protocol was not actually applied to HierCore, or the scores saturate under both protocols. Please clarify the exact procedure used for HierCore in each column and provide the underlying class-specific F1-max values.
- [Table 5] In the known-training condition, Image-level Diff. Ratio is 100.0% for all four datasets, with known and unknown F1-scores exactly equal (e.g., MVTec 97.9/97.9, VisA 92.2/92.2). Since known evaluation uses per-class optimal thresholds and unknown evaluation uses a single global threshold, exact equality requires an explanation or a correction of the experimental protocol.
minor comments (4)
- [Table 6] The table caption and text refer to memory bank construction time in hours, but the reported values (e.g., 317, 24+1, 1,886) appear to be in minutes; 1,886 minutes is about 31.4 hours, which the text later calls 'nearly 30 hours'. Please make the units consistent.
- [Appendix B tables] Several rows in Tables B.12, B.13, and B.14 have the strings 'Image' and 'Pixel' embedded inside the class-name column (e.g., 'Chewing gumImage', 'ConnectorImage'), which corrupts the table layout and makes the rows hard to parse.
- [Section 3.4, Equations (5)--(6)] The symbol P is used both for the patch extraction function in Equation (3) and for the total number of patches in Equations (5)--(6); please disambiguate. Also, the sampling ratio r is used in the complexity expressions but is not defined in the equations themselves.
- [Section 3.4] The worst-case complexity of FINCH is stated as O(N^2) in Equation (6), but no supporting derivation or citation is given for this bound; either provide a reference or remove the unverified term.
Circularity Check
HierCore's 100% Diff Ratio is an artifact of scoring the same threshold twice, making the label-robustness claim tautological.
-
fitted input called prediction
[Section 4.2 / Table 4 caption; Tables B.11-B.14; Section 3.3]
"During evaluation, F1-scores for known classes are computed using class-specific optimal thresholds, while a single optimal threshold is used for unknown-class evaluation."
For HierCore trained without labels, Table B.11 shows image-level F1 identical for Known and Unknown in every MVTec class (Bottle 99.2/99.2, Cable 96.3/96.3, ... Wood 96.7/96.7), and Table 4 reports Diff Ratio 100.0% on all datasets. A per-class optimal threshold cannot coincide with a single global threshold for all classes unless the same threshold was applied in both rows. HierCore's memory banks are built per semantic cluster, not per ground-truth class, so the class-specific threshold protocol is undefined for it, and Section 3.3's cluster-specific thresholds are never exercised because unknown evaluation uses one global threshold.
full rationale
The paper's central claim is that HierCore 'consistently meets all requirements,' especially Requirement 2 (comparable performance whether or not class labels are available during evaluation). The evidence for this is Table 4 (and Table 5), where HierCore's image-level F1 is exactly equal under 'Known' and 'Unknown' evaluation, yielding Diff Ratio 100.0%. The table caption defines the two conditions differently: class-specific optimal thresholds versus a single global optimal threshold. Exact equality across every class is unexplained and inconsistent with those different definitions. For HierCore, the 'Known' condition is also structurally problematic because the model groups data by semantic clusters rather than ground-truth classes, so computing class-specific thresholds requires an undefined mapping; the paper never describes how this was done. Meanwhile, the 'Unknown' condition uses one global threshold, which is the same scoring used for the 'Known' row in practice, making the comparison self-confirming. This is a fitted-input-called-prediction pattern: the robustness result is forced by using identical evaluation in both arms. There are no load-bearing self-citations and no imported uniqueness theorems; the efficiency comparisons and benchmark results are external and measurable. The main circularity concern is confined to the threshold-protocol artifact, but it is central because it underpins the 'satisfies all requirements' conclusion. Score 6 reflects that the central label-robustness prediction reduces by construction, even though the method itself has independent content.
Assumptions & free parameters
free parameters (3)
- Coreset sampling ratio =
0.10
- Feature layers for semantic and local features =
layer 4 semantic, layers 2-3 local
- Patch extraction window and stride =
3x3 window, stride 1
assumptions (3)
- domain assumption ImageNet-pretrained Wide-ResNet-50 features are semantically discriminative enough for FINCH clustering to recover meaningful groupings in industrial inspection data.
- ad hoc to paper Anomaly score distributions are comparable across semantic clusters, so a single global threshold is meaningful when class labels are absent.
- domain assumption Patch-level features from intermediate backbone layers contain local anomaly information, as in PatchCore.
Cite this review
Pith. "Pith review of Multi-class Image Anomaly Detection for Practical Applications: Requirements and Robust Solutions." pith.science (2026). https://pith.science/paper/PIKK7SLP
@misc{pith2026250802477,
author = {Pith},
title = {Pith review of: Multi-class Image Anomaly Detection for Practical Applications: Requirements and Robust Solutions},
year = {2026},
howpublished = {\url{https://pith.science/paper/PIKK7SLP}},
note = {Machine review of arXiv:2508.02477}
}
read the original abstract
Recent advances in image anomaly detection have extended unsupervised learning-based models from single-class settings to multi-class frameworks, aiming to improve efficiency in training time and model storage. When a single model is trained to handle multiple classes, it often underperforms compared to class-specific models in terms of per-class detection accuracy. Accordingly, previous studies have primarily focused on narrowing this performance gap. However, the way class information is used, or not used, remains a relatively understudied factor that could influence how detection thresholds are defined in multi-class image anomaly detection. These thresholds, whether class-specific or class-agnostic, significantly affect detection outcomes. In this study, we identify and formalize the requirements that a multi-class image anomaly detection model must satisfy under different conditions, depending on whether class labels are available during training and evaluation. We then re-examine existing methods under these criteria. To meet these challenges, we propose Hierarchical Coreset (HierCore), a novel framework designed to satisfy all defined requirements. HierCore operates effectively even without class labels, leveraging a hierarchical memory bank to estimate class-wise decision criteria for anomaly detection. We empirically validate the applicability and robustness of existing methods and HierCore under four distinct scenarios, determined by the presence or absence of class labels in the training and evaluation phases. The experimental results demonstrate that HierCore consistently meets all requirements and maintains strong, stable performance across all settings, highlighting its practical potential for real-world multi-class anomaly detection tasks.
Figures
Reference graph
Works this paper leans on
-
[1]
J. Liu, G. Xie, J. Wang, S. Li, C. Wang, F. Zheng, Y. Jin, Deep vi- sual anomaly detection in industrial manufacturing: A survey, arXiv preprint arXiv:2301.11514 1 (2023)
work page Pith review arXiv 2023
- [2]
-
[3]
T. Schlegl, P. Seeb¨ ock, S. M. Waldstein, G. Langs, U. Schmidt-Erfurth, f-anogan: Fast unsupervised anomaly detection with generative adversarial networks, Med- ical Image Analysis 54 (2019) 30–44
work page 2019
-
[4]
W. Sultani, C. Chen, M. Shah, Real-world anomaly detection in surveillance videos, in: CVPR, 2018, pp. 6479–6488
work page 2018
- [5]
-
[6]
K. K. Santhosh, D. P. Dogra, P. P. Roy, Anomaly detection in road traffic using visual surveillance: A survey, ACM Comput. Surv. 53 (6) (2020) 1–26
work page 2020
-
[7]
H. He, J. Zhang, H. Chen, X. Chen, Z. Li, X. Chen, Y. Wang, C. Wang, L. Xie, A diffusion-based framework for multi-class anomaly detection, in: AAAI, Vol. 38, 2024, pp. 8472–8480
work page 2024
- [8]
Show all 42 references
-
[9]
Zhang, C
J. Zhang, C. Wang, X. Li, G. Tian, Z. Xue, Y. Liu, G. Pang, D. Tao, Learning feature inversion for multi-class anomaly detection under general-purpose coco- ad benchmark, arXiv preprint arXiv:2404.10760 (2024)
2024 arXiv
-
[10]
H. He, Y. Bai, J. Zhang, Q. He, H. Chen, Z. Gan, C. Wang, X. Li, G. Tian, L. Xie, Mambaad: Exploring state space models for multi-class unsupervised anomaly detection, in: NeurIPS, 2024
2024
-
[11]
Jiang, Y
X. Jiang, Y. Chen, Q. Nie, J. Liu, Y. Liu, C. Wang, F. Zheng, Toward multi- class anomaly detection: Exploring class-aware unified model against inter-class interference, arXiv preprint arXiv:2403.14213 (2024)
2024 arXiv
-
[12]
Z. You, L. Cui, Y. Shen, K. Yang, X. Lu, Y. Zheng, X. Le, A unified model for multi-class anomaly detection, in: NeurIPS, Vol. 35, 2022, pp. 4571–4584
2022
-
[13]
L. He, Z. Jiang, J. Peng, W. Zhu, L. Liu, Q. Du, X. Hu, M. Chi, Y. Wang, C. Wang, Learning unified reference representation for unsupervised multi-class anomaly detection, in: ECCV, 2024, pp. 216–232
2024
-
[14]
Y. Lee, H. Lim, S. Jang, H. Yoon, Uniformaly: Towards task-agnostic uni- fied framework for visual anomaly detection, arXiv preprint arXiv:2307.12540 (2023)
2023 arXiv
-
[15]
Y. Zhou, X. Xu, Z. Sun, J. Song, A. Cichocki, H. T. Shen, Vq-flow: Tam- ing normalizing flows for multi-class anomaly detection via hierarchical vector quantization, arXiv preprint arXiv:2409.00942 (2024)
2024 arXiv
-
[16]
H. Song, M. Kim, D. Park, Y. Shin, J.-G. Lee, Learning from noisy labels with deep neural networks: A survey, IEEE Trans. Neural Netw. Learn. Syst. 34 (11) (2022) 8135–8153. 21
2022
-
[17]
D. Gong, L. Liu, V. Le, B. Saha, M. R. Mansour, S. Venkatesh, A. v. d. Hengel, Memorizing normality to detect anomaly: Memory-augmented deep autoen- coder for unsupervised anomaly detection, in: ICCV, 2019, pp. 1705–1714
2019
-
[18]
R. Lu, Y. Wu, L. Tian, D. Wang, B. Chen, X. Liu, R. Hu, Hierarchical vec- tor quantized transformer for multi-class unsupervised anomaly detection, in: NeurIPS, Vol. 36, 2023, pp. 8487–8500
2023
-
[19]
S. Lee, S. Lee, B. C. Song, Cfa: Coupled-hypersphere-based feature adaptation for target-oriented anomaly localization, IEEE Access 10 (2022) 78446–78454
2022
-
[20]
Defard, A
T. Defard, A. Setkov, A. Loesch, R. Audigier, Padim: a patch distribution mod- eling framework for anomaly detection and localization, in: ICPR Workshops, 2021, pp. 475–489
2021
-
[21]
K. Roth, L. Pemula, J. Zepeda, B. Sch¨ olkopf, T. Brox, P. Gehler, Towards total recall in industrial anomaly detection, in: CVPR, 2022, pp. 14318–14328
2022
-
[22]
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, L. Fei-Fei, Imagenet: A large-scale hierarchical image database, in: CVPR, 2009, pp. 248–255
2009
-
[23]
M. D. Zeiler, R. Fergus, Visualizing and understanding convolutional networks, in: ECCV, 2014, pp. 818–833
2014
-
[24]
Sarfraz, V
S. Sarfraz, V. Sharma, R. Stiefelhagen, Efficient parameter-free clustering using first neighbor relations, in: CVPR, 2019, pp. 8934–8943
2019
-
[25]
P. J. Rousseeuw, Silhouettes: a graphical aid to the interpretation and validation of cluster analysis, Journal of computational and applied mathematics 20 (1987) 53–65
1987
-
[26]
Sener, S
O. Sener, S. Savarese, Active learning for convolutional neural networks: A core-set approach, in: ICLR, 2018
2018
-
[27]
Zhang, H
J. Zhang, H. He, Z. Gan, Q. He, Y. Cai, Z. Xue, Y. Wang, C. Wang, L. Xie, Y. Liu, Ader: A comprehensive benchmark for multi-class visual anomaly de- tection, arXiv preprint arXiv:2406.03262 (2024)
2024 arXiv
-
[28]
Bergmann, M
P. Bergmann, M. Fauser, D. Sattlegger, C. Steger, Mvtec ad–a comprehensive real-world dataset for unsupervised anomaly detection, in: CVPR, 2019, pp. 9592–9600. 22
2019
-
[29]
Y. Zou, J. Jeong, L. Pemula, D. Zhang, O. Dabeer, Spot-the-difference self- supervised pre-training for anomaly detection and segmentation, in: ECCV, 2022, pp. 392–408
2022
-
[30]
Jezek, M
S. Jezek, M. Jonak, R. Burget, P. Dvorak, M. Skotak, Deep learning-based defect detection of metal parts: evaluating current methods in complex conditions, in: Int. Congr. Ultra Modern Telecommun. Control Syst. Workshops, IEEE, 2021, pp. 66–71
2021
-
[31]
Mishra, R
P. Mishra, R. Verk, D. Fornasier, C. Piciarelli, G. L. Foresti, Vt-adl: A vision transformer network for image anomaly detection and localization, in: IEEE Int. Symp. Ind. Electron., 2021, pp. 01–06
2021
-
[32]
Zavrtanik, M
V. Zavrtanik, M. Kristan, D. Skoˇ caj, Draem-a discriminatively trained recon- struction embedding for surface anomaly detection, in: ICCV, 2021, pp. 8330– 8339
2021
-
[33]
Z. Liu, Y. Zhou, Y. Xu, Z. Wang, Simplenet: A simple network for image anomaly detection and localization, in: CVPR, 2023, pp. 20402–20411
2023
-
[34]
Zhang, M
X. Zhang, M. Xu, X. Zhou, Realnet: A feature selection network with realistic synthetic anomaly for anomaly detection, in: CVPR, 2024, pp. 16699–16708
2024
-
[35]
H. Deng, X. Li, Anomaly detection via reverse distillation from one-class em- bedding, in: CVPR, 2022, pp. 9737–9746
2022
-
[36]
Z. Gu, J. Zhang, L. Liu, X. Chen, J. Peng, Z. Gan, G. Jiang, A. Shu, Y. Wang, L. Ma, Rethinking reverse distillation for multi-modal anomaly detection, in: AAAI, Vol. 38, 2024, pp. 8445–8453
2024
-
[37]
Zhang, S
X. Zhang, S. Li, X. Li, P. Huang, J. Shan, T. Chen, Destseg: Segmentation guided denoising student-teacher for anomaly detection, in: CVPR, 2023, pp. 3914–3923
2023
-
[38]
Gudovskiy, S
D. Gudovskiy, S. Ishizaka, K. Kozuka, Cflow-ad: Real-time unsupervised anomaly detection with localization via conditional normalizing flows, in: W ACV, 2022, pp. 98–107
2022
-
[39]
J. Lei, X. Hu, Y. Wang, D. Liu, Pyramidflow: High-resolution defect contrastive localization using pyramid normalizing flow, in: CVPR, 2023, pp. 14143–14152
2023
-
[40]
Zagoruyko, N
S. Zagoruyko, N. Komodakis, Wide residual networks, in: BMVC, 2016. 23
2016
-
[41]
Douze, A
M. Douze, A. Guzhva, C. Deng, J. Johnson, G. Szilvasy, P.-E. Mazar´ e, M. Lomeli, L. Hosseini, H. J´ egou, The faiss library, arXiv preprint arXiv:2401.08281 (2024)
2024 arXiv
-
[42]
Bergmann, M
P. Bergmann, M. Fauser, D. Sattlegger, C. Steger, Uninformed students: Student-teacher anomaly detection with discriminative latent embeddings, in: CVPR, 2020, pp. 4183–4192. Appendix A. Evaluation of Performance Gap between One-class and Multi- class Image Anomaly Detection T...
2020
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.