Pith. sign in

REVIEW 4 major objections 6 minor 20 references

HSS-IAD: A Heterogeneous Same-Sort Industrial Anomaly Detection Dataset

T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read By recombining five existing defect-image collections into 8,580 same-sort metal-part images, HSS-IAD cuts the average multiclass anomaly-detection AUROC from 94.0% on MVTec-AD to 67.0%, and pixel-level AUPRO from 83.2% to 43.8%.

desk verdict Useful same-sort IAD benchmark, but the headline AUROC drop is confounded by source-domain shift; the dataset is worth reviewing, the interpretation needs tightening. read the letter →

arxiv 2504.12689 v1 pith:Y7UNDVXV submitted 2025-04-17 cs.CV

Qishan Wang , Shuyong Gao , Junjie Hu , Jiawen Yu , Xuan Tong , You Li , Wenqiang Zhang This is my paper · ORCID
classification cs.CV
keywords HSS-IADindustrialanomalydetectionunsupervisedmulti-classsame-sortheterogeneitysurfacedefectlocalizationmetallicparts
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that current multiclass unsupervised anomaly detection (MUAD) benchmarks overstate real-world readiness because they mix product classes no single factory would make and use defects that are too obvious. To test the gap, it introduces HSS-IAD, a dataset of 8,580 images of metallic-like parts that are "same sort"—electrical commutators, magnetic tiles, steel sheet, engine castings—but vary in structure and appearance, with subtle defects that resemble machining marks, oil stains, and oxide scale. The paper reports that six reproduced state-of-the-art methods average a 94.0% image-level AUROC on MVTec-AD but only 67.0% on HSS-IAD, with pixel-level AUPRO falling from 83.2% to 43.8%, showing the dataset is markedly harder and better able to separate method strengths. It also provides foreground segmentations for synthetic anomaly generation, so the benchmark can be used for augmentation-based training as well as evaluation.

What carries the argument

The central object is the HSS-IAD dataset and its construction pipeline. Images are selected and reclassified from five existing defect datasets, deduplicated, and quality-filtered; castings are split into three categories defined by surface complexity and interference; steel-defect bounding boxes are converted into pixel masks; labeled regions that are too large, too small, or misclassified are manually corrected; and normal samples with confusable attached elements are deliberately kept. This curation is what creates the paper's intended test: the same-sort relationship supplies a coherent "normal" distribution, while structural variation, process features, and defect-material similarity force models to separate true anomalies from benign manufacturing variation. The authors also use a dichotomous image segmentation method to provide foreground masks, enabling synthetic anomaly generation for augmentation-based training.

What would settle it

Run the six methods on a matched control dataset built by randomly mixing the same five source collections without same-sort reclassification; if the AUROC drop relative to MVTec-AD is the same as on HSS-IAD, the challenge is explained by cross-dataset domain shift rather than same-sort factory realism. Collecting images from a single actual production line and comparing transfer from HSS-IAD-trained versus MVTec-AD-trained models would settle it directly.

Watch

Extended reading notes

Core claim

On the paper's own terms, the core discovery is that "same-sort heterogeneity" is a distinct failure mode for MUAD. HSS-IAD's seven categories are all metallic-like industrial parts, but within the same sort the appearance changes—castings have machined and unmachined surfaces, threaded holes, oil stains, oxide scale—and the defects are subtle enough to be mistaken for these normal features. When six unsupervised methods are evaluated under the multi-class protocol, the average image-level AUROC is 67.0% and the average pixel-level AUPRO is 43.8%, versus 94.0% and 83.2% on MVTec-AD; similar drops appear relative to VisA, Real-IAD, BTAD, and MPDD. The paper interprets this as evidence that the dataset captures real factory conditions that existing benchmarks miss, and that a more discriminative benchmark is needed to drive algorithmic progress.

Load-bearing premise

The dataset can stand in for a single factory's same-sort production line even though its images are drawn from five independent public datasets with different imaging conditions and were reorganized by manual reclassification and filtering.

Editorial extensions

If this is right

  • A model that scores well on existing MUAD benchmarks cannot be assumed ready for a real factory; the same methods lose roughly 27 points in image-level AUROC and 39 points in pixel-level AUPRO on HSS-IAD.
  • The high defect-to-background similarity and 3.0% anomalous-pixel ratio make HSS-IAD a stress test for small, subtle defect localization, so methods that rely on high-level features at the expense of spatial detail will be exposed.
  • Because foreground masks are provided, data-augmentation methods can be trained with synthetic anomalies on HSS-IAD, making the dataset useful both as an evaluation benchmark and as a training resource.
  • Method ranking on HSS-IAD changes relative to existing benchmarks, with feature- and reconstruction-based approaches outperforming latent-noise approaches, suggesting the benchmark can reveal which design choices matter for realistic industrial defects.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the same-sort heterogeneity is the active ingredient, then fine-tuning a method on HSS-IAD should transfer to a single factory's inspection line better than training on category-diverse datasets; the paper does not test this transfer, but it is a direct consequence of its motivation.
  • The drop in performance could partly come from cross-dataset domain shift (different cameras, lighting, resolutions) rather than same-sort difficulty, because the source images come from five independent collections; a control experiment mixing arbitrary categories from the same sources would separate these effects.
  • The provided foreground masks make it possible to generate synthetic anomalies of controlled subtlety; one could measure AUROC as a function of defect contrast or size to map exactly where current methods start failing, an experiment the paper does not run.
  • The class-separated versus multi-class gap suggests that unified MUAD models struggle mainly with learning one shared normal distribution across heterogeneous same-sort parts; class-conditional or prompt-guided normalization might close part of the gap.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper introduces HSS-IAD, a benchmark for multi-class unsupervised industrial anomaly detection (MUAD) containing 8,580 images of metallic-like parts organized into seven categories, assembled from five existing datasets (KolektorSDD2, Casting, MTD, KolektorSDD, and STEEL). The construction includes reclassification of Casting images into three finer categories, removal of blurry/duplicate/low-quality images, manual label filtering and refinement, and generation of pixel-level masks for STEEL; foreground masks are also released. The authors benchmark six methods (DeSTSeg, SimpleNet, DRÆM, UniAD, RD4AD, Dinomaly) under multi-class and class-separated settings, reporting image-level AUROC, pixel-level AP, and AUPRO. The central empirical claim is that HSS-IAD is much harder than existing benchmarks: average I-AUROC drops from 94.0% on MVTec-AD to 67.0% on HSS-IAD and P-AUPRO from 83.2% to 43.8% (Table II). The paper interprets this gap as evidence that the dataset captures same-sort structural/appearance variation and subtle, material-like defects found in real factories.

Significance. If the central claim is established, HSS-IAD fills a real need: most MUAD benchmarks mix semantically unrelated categories, while industrial settings require detecting anomalies across variants of the same product family. The paper's concrete strengths are a publicly released dataset, pixel-precise anomaly annotations, a reproducible benchmark protocol using official implementations, and results under two settings with detailed per-category tables. The reported performance gap is large and plausible. The main weakness is that the dataset is assembled from five independently collected source datasets, and the paper does not yet quantify whether the difficulty comes from same-sort realism or from cross-source acquisition shift; the 'same sort' claim is asserted rather than demonstrated with data. With additional diagnostic experiments and curation documentation, this could be a valuable benchmark.

major comments (4)
  1. [Section III.A, Table II] The headline result — average I-AUROC dropping from 94.0% on MVTec-AD to 67.0% on HSS-IAD and P-AUPRO from 83.2% to 43.8% — is presented as evidence that HSS-IAD is harder because of same-sort structural/appearance variation and subtle defects. However, HSS-IAD is built from five independently acquired datasets (KolektorSDD2, Casting, MTD, KolektorSDD, STEEL) with different cameras, lighting, resolutions, and backgrounds, and no control for cross-source domain shift is reported. A unified model could suffer or benefit from source-specific low-level cues, so the measured gap cannot yet be attributed to the claimed same-sort realism. Please add source-diagnostic experiments, for example per-source performance within HSS-IAD, source-controlled train/test splits, cross-source transfer tests, or low-level image statistics, and state explicitly which fraction of the performance drop persists when acquisition heterogeneity is controlled.
  2. [Sections I and III.A] The central motivation depends on the notions of 'same sort' products and 'the same factory', but neither is operationalized. The five sources are not shown to originate from a common production environment: steel sheets, magnetic tiles, electrical commutators, and castings can plausibly come from different plants and manufacturing processes. Please define 'same sort' with explicit inclusion criteria (e.g., shared material, product family, or manufacturing process), provide evidence that the selected categories can co-occur in one factory, or revise the claim to describe the dataset as 'metallic-like industrial parts from heterogeneous sources'.
  3. [Section III.A] The dataset construction relies on several qualitative curation steps — 'carefully evaluated' image quality, 'zero image check', 'manual label filtering', and reclassification of the Casting images — but no quantitative documentation is provided. For a dataset paper these steps are the method, and manual selection can bias both measured difficulty and the perceived similarity between defects and backgrounds. Please report the number of images removed or reclassified at each stage, the number and type of label corrections, annotator counts and agreement, and release the filtering/split lists or scripts so that the curation is reproducible.
  4. [Table I and Section III.B] Table I lists 'Similarity (between defect and background)' as a dataset attribute with values Low/High, and Section III.B states that the similarity in HSS-IAD is 'notably high', but no quantitative definition or measurement is given. This is a load-bearing attribute for the paper's claim that defects 'closely resemble the base materials'. Please define and report a reproducible similarity measure (e.g., low-level pixel statistics, feature-space distances, or human ratings) for HSS-IAD and for the comparison datasets.
minor comments (6)
  1. [Section III.B and Fig. 4] The text states that Fig. 4(c) shows 'the range of defect ratios for each class', but the caption describes Fig. 4(c) as the aspect ratio of the minimum bounding rectangles of defects; please align the text with the figure caption.
  2. [Fig. 6] In the caption of Fig. 6 the last panel is labeled '(e)', while the text refers to 'Fig. 6(c)' for the anomaly-localization visualizations; please make the panel labels consistent.
  3. [Section II] Section II contains the stray characters 'small.xs' at the end of the BTAD sentence; please remove them.
  4. [Table II] Table II leaves the DRÆM entry for Real-IAD blank without explanation; please state whether the method was omitted for resource reasons or could not be evaluated, since this affects the comparability of the Mean±Std row.
  5. [Tables II–IV] No information is given about the number of random seeds or runs used to produce the results; the reported 'Mean±Std' is across methods rather than across runs, which can be misread as run variability. Please state the experimental repetition protocol.
  6. [Abstract and Section III] The abstract and Section III mention foreground images for synthetic anomaly generation, but no experiment uses synthetic anomalies; please clarify explicitly that these masks are released for future use.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the benchmark difficulty is established empirically by external methods, not by construction or self-citation.

full rationale

This is a dataset-and-benchmark paper with no mathematical derivation chain to reduce. The central claim—that HSS-IAD is more challenging than existing IAD datasets—is supported by Table II, which reports image-level AUROC and pixel-level AUPRO of six existing methods (DeSTSeg, SimpleNet, DRÆM, UniAD, RD4AD, Dinomaly) reproduced from their official code with official hyperparameters. These are external algorithms evaluated on the new dataset, not quantities derived from the dataset definition. The dataset itself is assembled by selecting, filtering, and reclassifying images from five public sources (KolektorSDD2, Casting, MTD, KolektorSDD, STEEL); this curation is transparently described in Section III.A. Showing that a deliberately curated benchmark is hard is a property of the selection criteria, not circular reasoning. One mild point is that the authors intentionally included normal samples with confusing attached elements and subtle, material-like defects, so the observed difficulty is partly an intended consequence of curation; however, the paper does not define HSS-IAD 'difficulty' in terms of the benchmark results, nor does it fit any parameter and then call that fit a prediction. Reference [1] is the authors' own earlier casting dataset used as a source of images, but it is not invoked as proof of the benchmark's challenge; the challenge is demonstrated by the reproduced external methods. No self-citation chain is load-bearing. Therefore, no specific circular step can be quoted and exhibited, and the appropriate finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces no free parameters, no new theoretical entities, and no fitted constants. The central artifact is the dataset itself, which is an empirical construction rather than a postulated entity. The main assumptions are about the validity of the manual curation and the coherence of the source imagery.

assumptions (3)
  • domain assumption Manual reclassification and label refinement produce ground-truth annotations accurate enough for benchmarking.
    The dataset construction relies on the authors' manual inspection to remove 'excessive background noise', correct misclassified samples, and filter annotations (Section III-A). No inter-annotator agreement or external validation is reported.
  • domain assumption Images from five different public datasets can be treated as one coherent same-sort production environment.
    The core motivation is that products from one factory are same-sort but heterogeneous; yet the images come from KolektorSDD2, Casting, MTD, KolektorSDD and STEEL, which were collected independently. The paper does not show that imaging conditions are comparable.
  • standard math Standard anomaly detection metrics (I-AUROC, P-AP, P-AUPRO) are appropriate for comparing datasets.
    The benchmark relies on these metrics without discussing their limitations in the context of extreme class imbalance.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HSS-IAD: A Heterogeneous Same-Sort Industrial Anomaly Detection Dataset." pith.science (2026). https://pith.science/paper/Y7UNDVXV

@misc{pith2026250412689,
  author       = {Pith},
  title        = {Pith review of: HSS-IAD: A Heterogeneous Same-Sort Industrial Anomaly Detection Dataset},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Y7UNDVXV}},
  note         = {Machine review of arXiv:2504.12689}
}
read the original abstract

Multi-class Unsupervised Anomaly Detection algorithms (MUAD) are receiving increasing attention due to their relatively low deployment costs and improved training efficiency. However, the real-world effectiveness of MUAD methods is questioned due to limitations in current Industrial Anomaly Detection (IAD) datasets. These datasets contain numerous classes that are unlikely to be produced by the same factory and fail to cover multiple structures or appearances. Additionally, the defects do not reflect real-world characteristics. Therefore, we introduce the Heterogeneous Same-Sort Industrial Anomaly Detection (HSS-IAD) dataset, which contains 8,580 images of metallic-like industrial parts and precise anomaly annotations. These parts exhibit variations in structure and appearance, with subtle defects that closely resemble the base materials. We also provide foreground images for synthetic anomaly generation. Finally, we evaluate popular IAD methods on this dataset under multi-class and class-separated settings, demonstrating its potential to bridge the gap between existing datasets and real factory conditions. The dataset is available at https://github.com/Qiqigeww/HSS-IAD-Dataset.

Figures

Figures reproduced from arXiv: 2504.12689 by the authors.

Figure 1
Figure 1. Comparison of training paradigms. consist of various categories, such as fruits, snacks, pills, wine bottles, and fabrics, as shown in [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Normal and anomalous samples from existing datasets and the [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Data collection pipeline for our proposed HSS-IAD dataset. (a) The collection procedure encompasses various stages, including data collection, [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Statistical information of our proposed HSS-IAD dataset: (a) Distribution of anomaly/normal image quantities in training and test across different [PITH_FULL_IMAGE:figures/full_fig_p003_4.png]
Figure 5
Figure 5. Figure 5: (a) Casting image; (b) Foreground image of Casting. [PITH_FULL_IMAGE:figures/full_fig_p003_5.png]
Figure 6
Figure 6. Figure 6: Qualitative results for anomaly localization on HSS-IAD dataset. (a) [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 17 canonical work pages

  1. [1]

    A casting surface dataset and benchmark for subtle and confusable defect detection in complex contexts,

    Qishan Wang, Shuyong Gao, Li Xiong, Aili Liang, Kaidong Jiang, and Wenqiang Zhang, “A casting surface dataset and benchmark for subtle and confusable defect detection in complex contexts,” IEEE Sensors Journal, 2024

    work page 2024

  2. [2]

    A unified model for multi-class anomaly detection,

    Zhiyuan You, Lei Cui, Yujun Shen, Kai Yang, Xin Lu, Yu Zheng, and Xinyi Le, “A unified model for multi-class anomaly detection,” Advances in Neural Information Processing Systems , vol. 35, pp. 4571– 4584, 2022

    work page 2022

  3. [3]

    Exploring plain vit reconstruction for multi-class unsupervised anomaly detection,

    Jiangning Zhang, Xuhai Chen, Yabiao Wang, Chengjie Wang, Yong Liu, Xiangtai Li, Ming-Hsuan Yang, and Dacheng Tao, “Exploring plain vit reconstruction for multi-class unsupervised anomaly detection,” arXiv preprint arXiv:2312.07495, 2023

    arXiv 2023

  4. [4]

    Mvtec ad–a comprehensive real-world dataset for unsupervised anomaly detection,

    Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Ste- ger, “Mvtec ad–a comprehensive real-world dataset for unsupervised anomaly detection,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2019, pp. 9592–9600

    work page 2019

  5. [5]

    Real-iad: A real-world multi-view dataset for benchmarking versatile industrial anomaly detection,

    Chengjie Wang, Wenbing Zhu, Bin-Bin Gao, Zhenye Gan, Jiangning Zhang, Zhihao Gu, Shuguang Qian, Mingang Chen, and Lizhuang Ma, “Real-iad: A real-world multi-view dataset for benchmarking versatile industrial anomaly detection,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 22883–22892

    work page 2024

  6. [6]

    Beyond dents and scratches: Logical constraints in unsupervised anomaly detection and localization,

    Paul Bergmann, Kilian Batzner, Michael Fauser, David Sattlegger, and Carsten Steger, “Beyond dents and scratches: Logical constraints in unsupervised anomaly detection and localization,” International Journal of Computer Vision , vol. 130, no. 4, pp. 947–969, 2022

    work page 2022

  7. [7]

    Deep industrial image anomaly detection: A survey,

    Jiaqi Liu, Guoyang Xie, Jingbao Wang, Shangnian Li, Chengjie Wang, Feng Zheng, and Yaochu Jin, “Deep industrial image anomaly detection: A survey,” arXiv preprint arXiv:2301.11514 , 2023

    arXiv 2023

  8. [8]

    Destseg: Segmentation guided denoising student-teacher for anomaly detection,

    Xuan Zhang, Shiyu Li, Xi Li, Ping Huang, Jiulong Shan, and Ting Chen, “Destseg: Segmentation guided denoising student-teacher for anomaly detection,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 3914–3923

    work page 2023

Show all 20 references
  1. [9]

    Draem-a dis- criminatively trained reconstruction embedding for surface anomaly detection,

    Vitjan Zavrtanik, Matej Kristan, and Danijel Sko ˇcaj, “Draem-a dis- criminatively trained reconstruction embedding for surface anomaly detection,” in Proceedings of the IEEE/CVF international conference on computer vision , 2021, pp. 8330–8339

    2021

  2. [10]

    Highly accurate dichotomous image segmentation,

    Xuebin Qin, Hang Dai, Xiaobin Hu, Deng-Ping Fan, Ling Shao, and Luc Van Gool, “Highly accurate dichotomous image segmentation,” in European Conference on Computer Vision . Springer, 2022, pp. 38–56

    2022

  3. [11]

    Segmentation-based deep-learning approach for surface-defect detec- tion,

    Domen Tabernik, Samo ˇSela, Jure Skvar ˇc, and Danijel Sko ˇcaj, “Segmentation-based deep-learning approach for surface-defect detec- tion,” Journal of Intelligent Manufacturing , vol. 31, no. 3, pp. 759–776, 2020

    2020

  4. [12]

    Mixed supervision for surface-defect detection: From weakly to fully supervised learning,

    Jakob Bo ˇziˇc, Domen Tabernik, and Danijel Sko ˇcaj, “Mixed supervision for surface-defect detection: From weakly to fully supervised learning,” Computers in Industry , vol. 129, pp. 103459, 2021

    2021

  5. [13]

    Surface defect saliency of magnetic tile,

    Yibin Huang, Congying Qiu, and Kui Yuan, “Surface defect saliency of magnetic tile,” The Visual Computer , vol. 36, no. 1, pp. 85–96, 2020

    2020

  6. [14]

    Severstal: Steel defect detection,

    Alexey Grishin, BorisV , iBardintsev, inversion, and Oleg, “Severstal: Steel defect detection,” https://kaggle.com/competitions/ severstal-steel-defect-detection, 2019, Kaggle

    2019

  7. [15]

    Deep learning-based defect detection of metal parts: evaluating current methods in complex conditions,

    Stepan Jezek, Martin Jonak, Radim Burget, Pavel Dvorak, and Milos Skotak, “Deep learning-based defect detection of metal parts: evaluating current methods in complex conditions,” in 2021 13th International congress on ultra modern telecommunications and control systems and wor...

    2021

  8. [16]

    Spot-the-difference self-supervised pre-training for anomaly detection and segmentation,

    Yang Zou, Jongheon Jeong, Latha Pemula, Dongqing Zhang, and Onkar Dabeer, “Spot-the-difference self-supervised pre-training for anomaly detection and segmentation,” in European Conference on Computer Vision. Springer, 2022, pp. 392–408

    2022

  9. [17]

    Vt-adl: A vision transformer network for image anomaly detection and localization,

    Pankaj Mishra, Riccardo Verk, Daniele Fornasier, Claudio Piciarelli, and Gian Luca Foresti, “Vt-adl: A vision transformer network for image anomaly detection and localization,” in 2021 IEEE 30th International Symposium on Industrial Electronics (ISIE) . IEEE, 2021, pp. 01–06

    2021

  10. [18]

    Simplenet: A simple network for image anomaly detection and localization,

    Zhikang Liu, Yiming Zhou, Yuansheng Xu, and Zilei Wang, “Simplenet: A simple network for image anomaly detection and localization,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 20402–20411

    2023

  11. [19]

    Anomaly detection via reverse distillation from one-class embedding,

    Hanqiu Deng and Xingyu Li, “Anomaly detection via reverse distillation from one-class embedding,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2022, pp. 9737–9746

    2022

  12. [20]

    Dinomaly: The less is more philosophy in multi-class unsupervised anomaly detection,

    Jia Guo, Shuai Lu, Weihang Zhang, Fang Chen, Hongen Liao, and Huiqi Li, “Dinomaly: The less is more philosophy in multi-class unsupervised anomaly detection,” arXiv preprint arXiv:2405.14325 , 2024

    2024 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.