REVIEW 3 major objections 6 minor 28 references
Detect, Classify, Act: Categorizing Industrial Anomalies with Multi-Modal Large Language Models
T0 review · 3 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read A two-stage pipeline—fast anomaly detector plus multimodal LLM—classifies industrial defects at 84 percent accuracy without task-specific training.
desk verdict The pipeline idea is genuinely useful, but the headline accuracy numbers don't match the paper's own metric equation, so the SOTA claim is uninterpretable until they clarify what was computed. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the division of labor between a fast pixel-level anomaly detector and a multimodal LLM, joined by two kinds of prompts. The visual prompt is a red-line contour overlay on the localized anomaly, presented next to the query image and a normal reference image; the text prompt is structured into a normal-object description, anomaly class definitions, and an explicit classification strategy. The detector's job is to keep classification cheap and accurate by filtering normal samples and telling the LLM where to look, while the LLM's job is to apply semantic knowledge and follow user-defined class definitions. Ablation results show that removing the anomaly descriptions causes the largest accuracy drop, and that both the reference image and the contour overlay contribute; the complete prompt gives the best accuracy, supporting the claim that both visual and textual context matter.
What would settle it
Have independent annotators re-label a random subset of MVTec-AC and VisA-AC from the original images using the paper's class definitions and compare their labels with the published ones; low inter-annotator agreement or systematic disagreement with the original dataset labels would mean the reported accuracy figures are not a valid measure of classification performance.
Extended reading notes
Core claim
The central claim is that a specialized localization module and a semantically rich language model can be composed into a classifier that needs no training for the classification task. In VELM, a pixel-level anomaly detector acts as the Vision Expert: it decides whether an image is normal, and when it detects an anomaly it produces an anomaly map. That map is converted into a visual prompt by drawing a red contour around the detected region, and the LLM receives the query image, the contour image, a normal reference image, and a structured text prompt containing a normal-object description, definitions of every anomaly class, and a classification strategy. The paper reports that on MVTec-AD this composition reaches 81.4 percent accuracy, outperforming prior anomaly classification methods by 5 percentage points; on the refined MVTec-AC benchmark it reaches 84.0 percent with the DDAD detector and 87.8 percent when ground-truth masks are used instead. The authors further claim the same pipeline separates benign anomalies from critical defects with 89.8 percent accuracy in a simulated three-way normal/anomaly/defect task.
Load-bearing premise
The headline accuracy numbers rest on the paper's own manual relabeling of MVTec-AD and VisA—36 corrected samples, merged classes, and removal of small classes—being the correct ground truth, and no independent annotator or external audit checks those labels.
Editorial extensions
If this is right
- Anomaly classification becomes a zero-shot task: with an off-the-shelf detector and a pretrained multimodal LLM, an inspection system can name defect types without collecting labeled defect examples for each new object.
- Classification accuracy is bounded by localization quality; the gap between oracle masks (87.8 percent) and real detectors (84.0 percent for DDAD, 78.1 percent for PatchCore) implies that better anomaly segmentation directly raises classification performance.
- Prompt structure is a real component of the method; removing anomaly descriptions lowers accuracy by about five points, so the way classes are defined in language is part of the approach, not a detail.
- A single pipeline can triage anomalies into normal, negligible anomaly, and critical defect at 89.8 percent accuracy in the paper's simulated setting, which is the decision problem industrial inspection actually faces.
- MVTec-AC and VisA-AC offer a benchmark for classification rather than detection, and their corrected and merged labels can support comparable evaluation of future methods.
Reading between the lines
- One implication left implicit is that the random choice of a normal reference image introduces variance in the reported numbers; a natural extension is to evaluate across multiple reference images per query or to learn which reference best supports the prompt.
- If the dependency on localization quality holds generally, then the practical bottleneck for new industrial objects is the detector, not the language model; swapping in a detector specialized per object family should be tested as a way to close the gap to the oracle result.
- The closed-set limitation named in the paper suggests a next step: using the LLM's open vocabulary to output a description when the anomaly fits no user-defined class, which would measure how much the semantic model adds beyond the predefined labels.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes VELM, a two-stage pipeline for industrial anomaly classification that combines a fast unsupervised anomaly detector (DDAD or PatchCore) with a multimodal LLM (GPT-4o or GPT-4o-mini). When the detector flags an anomaly, the LLM receives a normal reference image, the query image, a red-contour visual prompt derived from the detector's localization, and a structured text prompt that defines the anomaly classes and a classification strategy. The authors also introduce two refined datasets, MVTec-AC and VisA-AC, with corrected, merged, and filtered anomaly class labels, and evaluate VELM on these and on the original MVTec-AD. They report state-of-the-art accuracy on MVTec-AD (81.4% in the body, 80.4% in the abstract) and strong performance on MVTec-AC (84.0% with DDAD+GPT-4o), together with ablations of the prompt components and a simulated anomaly-vs-defect experiment.
Significance. The two-stage design is practically motivated: using a fast detector to filter normal images and to localize anomalies before invoking the LLM is a sensible way to balance speed and semantic flexibility. The ablations in Table 5 clearly show the value of the reference image, visual prompt, and text descriptions. The paper provides a code link and proposes concrete benchmark refinements that could be useful to the community. However, the central quantitative claims depend on an apparently mis-defined evaluation metric and on benchmarks whose labels are authored by the same researchers without external validation or a public data release. Until these issues are resolved, the reported accuracy numbers and the claimed state of the art are not interpretable.
major comments (3)
- [§4.1, Eq. (1)] The 'macro accuracy' defined in Eq. (1) is not an accuracy measure; it is the mean per-class Jaccard index (intersection over union), because the denominator TP_c + FP_c + FN_c is the union of ground-truth and predicted positive images for class c and excludes true negatives. In a single-label multi-class setting, this quantity is far lower than conventional accuracy for the same predictions; for a balanced three-class problem with 84% per-class recall and precision, Eq. (1) would give about 0.72, and with realistic zero-shot precision it would be substantially lower. The values reported in Tables 2 and 3 (e.g., 84.0 and 87.6) are therefore not reproducible from the published formula. Please clarify the actual computed metric (e.g., macro recall or overall accuracy), correct the equation, or recompute every headline number. Since the abstract and Tables 1–3 all depend on this quantity, the central SOTA claim is currently uninterpretable.
- [§3.3, Tables 2–3, Figure 4] The MVTec-AC and VisA-AC benchmarks are central contributions, but their labels are the authors' manual corrections (36 corrected samples, merged classes, removed low-support classes) with no inter-annotator agreement, no external audit, and no public dataset link; the paper only provides a code repository. Moreover, the text prompts used for classification (Figure 4) encode the same class definitions and classification strategy that the authors used to assign the corrected labels. This creates a circularity risk: the LLM may be following the label-generation rules rather than independently recognizing visual anomalous patterns. Please release the datasets and a detailed annotation protocol, report inter-annotator statistics, and discuss how the evaluation avoids this circularity, for example by holding out a test set labeled by independent experts.
- [§4.3, Table 4] The anomaly-vs-defect experiment does not provide a valid measure of the claimed capability. The 'Normal' column is trivially 1.0 for all but one object category because normal images are filtered before the LLM; the random designation of 30% of anomaly classes as 'negligible' conflates class-level severity with instance-level acceptability; and the 89.8% mean accuracy aggregates this trivial component with the two-class anomaly/defect split. Please redesign the experiment with instance-level severity labels and a proper three-way evaluation, or clearly restrict the claim to the distinction between the two anomaly groups.
minor comments (6)
- [Abstract vs. §4.2.1, Table 1] The reported accuracy on MVTec-AD is 80.4% in the abstract and 81.4% in Section 4.2.1 and Table 1; please correct the inconsistency and verify the margin over Echo, since 81.4 − 72.9 = 8.5, not 9.5.
- [§3.3] The paper gives no public link for the MVTec-AC and VisA-AC datasets, only the code repository; please provide a dataset release to enable reproducibility and independent verification.
- [§4.2] The evaluation randomly selects a reference normal image from the training set, but the random seed is not reported; since the results may depend on this choice, please report seeds or average over multiple selections.
- [Table 1] F1-score and Cohen's kappa are reported for VELM but not for the baseline methods Echo and MCAD; please add these metrics or state why they are unavailable.
- [§3.3.1] Please specify which four 'combined' anomaly classes were excluded and list the exact per-object class merges, so that the benchmark construction is fully reproducible.
- [§5] The phrase 'first anomaly classification framework' is too strong, as prior work (MCAD, Echo) already addresses anomaly classification; please soften this claim.
Circularity Check
No significant circularity: VELM's anomaly-class output comes from an off-the-shelf multimodal LLM with no task-specific training; the author-provided DDAD/PatchCore detectors are loaded as external tools, and the classification numbers are not forced by construction.
full rationale
The paper makes no parameter fitting that would make a reported number equivalent to its input. The classification step is performed by GPT-4o/GPT-4o-mini on images plus prompts; the vision expert only decides whether to invoke the LLM and provides a red-contour localization. None of the evaluation labels are used to fit weights or prompts, and ablations show the classification accuracy depends on the detector (Oracle 87.8, DDAD 84.0, PatchCore 78.1) rather than being fixed by any single input. The self-citations to DDAD and PatchCore are standard uses of prior detectors, not load-bearing uniqueness claims: the paper also evaluates with ground-truth masks and with GPT-4o-mini, so the claimed numbers do not reduce to a self-citation chain. The MVTec-AC/VisA-AC labels are author-created, but prompt definitions of anomaly classes are descriptions of class semantics, not per-image answer keys; without evidence that the prompts encode the specific corrected labels, no definitional equivalence can be exhibited. The main internal inconsistency is metric definition: Eq. (1) in Section 4.1 defines mean per-class Jaccard (TP/(TP+FP+FN)), not accuracy, yet the tables report the resulting numbers as 'Acc'. This makes the headline numbers hard to interpret and would be a correctness/reproducibility issue, but it is not a circular derivation of a prediction from its own input.
Assumptions & free parameters
free parameters (3)
- MVTec-AC class merge/exclusion policy =
4 merged pairs; toothbrush removed; 4 combined classes excluded
- VisA-AC minimum class size threshold =
10 samples
- Per-category LLM prompt contents =
hand-written normal descriptions, anomaly definitions, classification strategies
assumptions (4)
- domain assumption GPT-4o can visually ground industrial close-up images well enough to map the supplied rules to pixels
- domain assumption The vision expert (DDAD) localizes anomalies well enough that red-contour prompts help rather than mislead
- domain assumption The authors' manual corrections and merges of MVTec-AD and VisA labels are ground truth
- ad hoc to paper Equation (1) is a valid accuracy measure
invented entities (2)
-
MVTec-AC
-
VisA-AC
Cite this review
Pith. "Pith review of Detect, Classify, Act: Categorizing Industrial Anomalies with Multi-Modal Large Language Models." pith.science (2026). https://pith.science/paper/N44RVE35
@misc{pith2026250502626,
author = {Pith},
title = {Pith review of: Detect, Classify, Act: Categorizing Industrial Anomalies with Multi-Modal Large Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/N44RVE35}},
note = {Machine review of arXiv:2505.02626}
}
read the original abstract
Recent advances in visual industrial anomaly detection have demonstrated exceptional performance in identifying and segmenting anomalous regions while maintaining fast inference speeds. However, anomaly classification-distinguishing different types of anomalies-remains largely unexplored despite its critical importance in real-world inspection tasks. To address this gap, we propose VELM, a novel LLM-based pipeline for anomaly classification. Given the critical importance of inference speed, we first apply an unsupervised anomaly detection method as a vision expert to assess the normality of an observation. If an anomaly is detected, the LLM then classifies its type. A key challenge in developing and evaluating anomaly classification models is the lack of precise annotations of anomaly classes in existing datasets. To address this limitation, we introduce MVTec-AC and VisA-AC, refined versions of the widely used MVTec-AD and VisA datasets, which include accurate anomaly class labels for rigorous evaluation. Our approach achieves a state-of-the-art anomaly classification accuracy of 80.4% on MVTec-AD, exceeding the prior baselines by 5%, and 84% on MVTec-AC, demonstrating the effectiveness of VELM in understanding and categorizing anomalies. We hope our methodology and benchmark inspire further research in anomaly classification, helping bridge the gap between detection and comprehensive anomaly characterization.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,
-
[2]
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei 8 Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023. 3
arXiv 2023
-
[3]
Efficien- tad: Accurate visual anomaly detection at millisecond-level latencies
Kilian Batzner, Lars Heckler, and Rebecca K ¨onig. Efficien- tad: Accurate visual anomaly detection at millisecond-level latencies. In Proceedings of the IEEE/CVF Winter Confer- ence on Applications of Computer Vision , pages 128–138,
-
[4]
Mvtec ad–a comprehensive real-world dataset for unsupervised anomaly detection
Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger. Mvtec ad–a comprehensive real-world dataset for unsupervised anomaly detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9592–9600, 2019. 3, 4
work page 2019
-
[5]
Grounding everything: Emerging localiza- tion properties in vision-language transformers
Walid Bousselham, Felix Petersen, Vittorio Ferrari, and Hilde Kuehne. Grounding everything: Emerging localiza- tion properties in vision-language transformers. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3828–3837, 2024. 3
2024
-
[6]
Clip-ad: A language-guided staged dual- path model for zero-shot anomaly detection
Xuhai Chen, Jiangning Zhang, Guanzhong Tian, Haoyang He, Wuhao Zhang, Yabiao Wang, Chengjie Wang, Yunsheng Wu, and Yong Liu. Clip-ad: A language-guided staged dual- path model for zero-shot anomaly detection. arXiv preprint arXiv:2311.00453, 2023. 3
arXiv 2023
-
[7]
Zhiling Chen, Hanning Chen, Mohsen Imani, and Farhad Imani. Can multimodal large language models be guided to improve industrial anomaly detection? arXiv preprint arXiv:2501.15795, 2025. 2, 3, 6
arXiv 2025
-
[8]
Sub-image anomaly detec- tion with deep pyramid correspondences
Niv Cohen and Yedid Hoshen. Sub-image anomaly detec- tion with deep pyramid correspondences. arXiv preprint arXiv:2005.02357, 2020. 2
arXiv 2005
Show all 28 references
-
[9]
Padim: a patch distribution modeling framework for anomaly detection and localization
Thomas Defard, Aleksandr Setkov, Angelique Loesch, and Romaric Audigier. Padim: a patch distribution modeling framework for anomaly detection and localization. In Inter- national conference on pattern recognition, pages 475–489. Springer, 2021
2021
-
[10]
Anomaly detection via reverse distillation from one-class embedding
Hanqiu Deng and Xingyu Li. Anomaly detection via reverse distillation from one-class embedding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9737–9746, 2022. 2
2022
-
[11]
Visual prompt engineering for medical vision language models in radiology
Stefan Denner, Markus Bujotzek, Dimitrios Bounias, David Zimmerer, Raphael Stock, Paul F J ¨ager, and Klaus Maier- Hein. Visual prompt engineering for medical vision language models in radiology. arXiv preprint arXiv:2408.15802 ,
-
[12]
Diffusion for out-of-distribution detection on road scenes and beyond
Silvio Galesso, Philipp Schr ¨oppel, Hssan Driss, and Thomas Brox. Diffusion for out-of-distribution detection on road scenes and beyond. In European Conference on Computer Vision, pages 110–126. Springer, 2024. 1
2024
-
[13]
Anomalygpt: Detecting in- dustrial anomalies using large vision-language models
Zhaopeng Gu, Bingke Zhu, Guibo Zhu, Yingying Chen, Ming Tang, and Jinqiao Wang. Anomalygpt: Detecting in- dustrial anomalies using large vision-language models. In Proceedings of the AAAI conference on artificial intelli- gence, pages 1932–1940, 2024. 3
1932
-
[14]
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. 2
2016
-
[15]
Denoising dif- fusion probabilistic models
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. 2
2020
-
[16]
Winclip: Zero- /few-shot anomaly classification and segmentation
Jongheon Jeong, Yang Zou, Taewan Kim, Dongqing Zhang, Avinash Ravichandran, and Onkar Dabeer. Winclip: Zero- /few-shot anomaly classification and segmentation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19606–19616, 2023. 2, 3
2023
-
[17]
Mmad: The first-ever comprehensive benchmark for mul- timodal large language models in industrial anomaly detec- tion
Xi Jiang, Jian Li, Hanqiu Deng, Yong Liu, Bin-Bin Gao, Yifeng Zhou, Jialin Li, Chengjie Wang, and Feng Zheng. Mmad: The first-ever comprehensive benchmark for mul- timodal large language models in industrial anomaly detec- tion. arXiv preprint arXiv:2410.09453, 2024. 2, 3
-
[18]
Myriad: Large multimodal model by applying vi- sion experts for industrial anomaly detection
Yuanze Li, Haolin Wang, Shihao Yuan, Ming Liu, Debin Zhao, Yiwen Guo, Chen Xu, Guangming Shi, and Wang- meng Zuo. Myriad: Large multimodal model by applying vi- sion experts for industrial anomaly detection. arXiv preprint arXiv:2310.19070, 2023. 3
-
[19]
Mcad: Multi- classification anomaly detection with relational knowledge distillation
Zhuo Li, Yifei Ge, Xuebin Yue, and Lin Meng. Mcad: Multi- classification anomaly detection with relational knowledge distillation. Neural Computing and Applications , 36(23): 14543–14557, 2024. 3, 6
2024
-
[20]
Anomaly detection with conditioned denoising diffusion models
Arian Mousakhan, Thomas Brox, and Jawad Tayyub. Anomaly detection with conditioned denoising diffusion models. arXiv preprint arXiv:2305.15956 , 2023. 1, 2, 4, 6
2023 arXiv
-
[21]
Henriques, and Fatma G¨uney
Nazir Nayal, Mısra Yavuz, Jo ˜ao F. Henriques, and Fatma G¨uney. Rba: Segmenting unknown regions rejected by all. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023. 1
2023
-
[22]
Learning transferable visual models from natural language supervi- sion
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...
2021
-
[23]
Towards to- tal recall in industrial anomaly detection
Karsten Roth, Latha Pemula, Joaquin Zepeda, Bernhard Sch¨olkopf, Thomas Brox, and Peter Gehler. Towards to- tal recall in industrial anomaly detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14318–14328, 2022. 1, 6
2022
-
[24]
Score-based generative modeling through stochastic differential equa- tions
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Ab- hishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equa- tions. arXiv preprint arXiv:2011.13456, 2020. 2
2011 arXiv
-
[25]
Diffusion models for medical anomaly detection
Julia Wolleb, Florentin Bieder, Robin Sandk ¨uhler, and Philippe C Cattin. Diffusion models for medical anomaly detection. In International Conference on Medical image computing and computer-assisted intervention, pages 35–45. Springer, 2022. 1
2022
-
[26]
Unsupervised surface anomaly detec- tion with diffusion probabilistic model
Xinyi Zhang, Naiqi Li, Jiawei Li, Tao Dai, Yong Jiang, and Shu-Tao Xia. Unsupervised surface anomaly detec- tion with diffusion probabilistic model. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 6782–6791, 2023. 2
2023
-
[27]
Anomalyclip: Object-agnostic prompt learn- 9 ing for zero-shot anomaly detection
Qihang Zhou, Guansong Pang, Yu Tian, Shibo He, and Jiming Chen. Anomalyclip: Object-agnostic prompt learn- 9 ing for zero-shot anomaly detection. arXiv preprint arXiv:2310.18961, 2023. 3
2023
-
[28]
Spot-the-difference self-supervised pre- training for anomaly detection and segmentation
Yang Zou, Jongheon Jeong, Latha Pemula, Dongqing Zhang, and Onkar Dabeer. Spot-the-difference self-supervised pre- training for anomaly detection and segmentation. In Eu- ropean Conference on Computer Vision , pages 392–408. Springer, 2022. 3, 5 10
2022
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.