Pith. sign in

REVIEW 5 major objections 7 minor 25 references

Automatic Prompt Generation and Grounding Object Detection for Zero-Shot Image Anomaly Detection

T0 review · 5 major / 7 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A zero-shot pipeline combining GPT-3 prompts, Grounding DINO crops, and CLIP scoring beats WinCLIP on industrial anomaly detection.

desk verdict Plausible incremental win on zero-shot anomaly detection, but the key prompt-generation step is a black box and the paper's own conclusion describes a different method; needs reporting fixes before I trust the numbers. read the letter →

arxiv 2411.19220 v1 pith:QJRVE4DL submitted 2024-11-28 cs.CV cs.MM

Tsun-Hin Cheung , Ka-Chun Fung , Songjiang Lai , Kwan-Ho Lin , Vincent Ng , Kin-Man Lam This is my paper · ORCID
classification cs.CVcs.MM
keywords zero-shotanomalydetectionindustrialinspectionpromptgenerationgroundingobjectCLIPGPT-3MVTec-ADVisA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that industrial defect detection can be done without any training data or fine-tuning, by chaining three pretrained models: GPT-3 writes text prompts for what a normal and an anomalous product look like, Grounding DINO crops the product out of the image, and CLIP scores the crop against those prompts. The authors report that this training-free pipeline reaches an AUROC of 93.2% and an AUPR of 96.6% on the MVTec-AD benchmark, and 82.9% and 85.7% on VisA, both above the zero-shot WinCLIP baseline. The point of the approach is that a factory could set up inspection for a new product just by naming its category, without collecting labeled defect examples or training a model. If the reported results hold, it would make quality control cheaper, faster, and easier to deploy across many product lines.

What carries the argument

The load-bearing mechanism is a three-stage zero-shot pipeline. GPT-3, a large language model, produces the text prompts; Grounding DINO, a text-guided open-set object detector, produces a bounding box and crop of the product; and CLIP, a contrastive image-text model, embeds both text and image and computes the anomaly score via the softmax ratio in Equation (4). The averaging of the global image embedding with the local object-crop embedding is what fuses context with product-specific detail, and the anomaly score formula converts the two prompt similarities into a single probability-like number.

What would settle it

Replace the GPT-3 prompt sets in the published method with WinCLIP's fixed template prompts while keeping the Grounding DINO crop and CLIP scoring; if the AUROC on MVTec-AD does not fall below the reported 93.2%, the prompt-generation component is not the source of the gain and the claim would be falsified. Symmetrically, feed the same GPT-3 prompts to CLIP on whole images without the object crop; if AUROC stays near 93.2%, the grounding component is not load-bearing.

Watch

Extended reading notes

Core claim

The central claim is that automatic prompt generation and object grounding together improve zero-shot image anomaly detection over the existing CLIP-based approaches. Instead of relying on fixed human-designed template prompts, the method lets GPT-3 generate two sets of product-specific prompts, $P_{\text{normal}}$ and $P_{\text{anomaly}}$, from inputs $x_{\text{normal}}$ and $x_{\text{anomaly}}$. Grounding DINO then detects the product in the image and crops it to $I_{\text{object}}$, and CLIP computes an anomaly score as the ratio of the fused image embedding's similarity to the anomaly prompt over its similarities to both prompts: $s = (e_{\text{fused}} \cdot t_{\text{anomaly}}) / (e_{\text{fused}} \cdot t_{\text{anomaly}} + e_{\text{fused}} \cdot t_{\text{normal}})$, where $e_{\text{fused}}$ averages the whole-image and object-crop embeddings. The paper reports state-of-the-art zero-shot results on MVTec-AD and VisA, and ablation studies show that removing either the prompt generation or the object detection component degrades performance, supporting the claim that both modules contribute to the gain.

Load-bearing premise

The whole pipeline depends on the unstated inputs $x_{\text{normal}}$ and $x_{\text{anomaly}}$ and on GPT-3 turning them into sensible, object-specific prompt sets, yet the paper never reports these inputs, the prompts, or any quality check, so the anomaly score's numerator and denominator rest on an unvalidated black-box language model.

Editorial extensions

If this is right

  • A factory could set up defect screening for a new product by supplying only the product category name, with no labeled defect images and no model training.
  • The reported margin over WinCLIP on both MVTec-AD and VisA suggests that class-specific LLM prompts and object-cropped features carry information that fixed templates and sliding windows miss.
  • The ablation results, showing AUROC drops of roughly 1 to 3 points when either module is removed, indicate that both prompt generation and object localization are needed for the full gain.
  • Because the pipeline is built from three separate pretrained models, each component can be upgraded independently, so future improvements in LLM prompt quality or open-set detection could translate directly into better anomaly detection.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper never reports the exact inputs $x_{\text{normal}}$, $x_{\text{anomaly}}$ or the generated prompt sets, so the method as published is not fully reproducible; releasing these would let others verify that the gain comes from the LLM prompts rather than from the object detector alone.
  • A natural test of the mechanism's source of gain would be to compare the reported improvements across MVTec-AD categories of different object sizes; if most of the improvement concentrates in categories where the product occupies a small part of the image, the crop step is doing the work.
  • The same prompt-and-ground architecture could transfer to other inspection domains, such as detecting anomalies in a specific organ in medical scans or in a known structure in satellite imagery, whenever a class name is available for grounding.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The paper proposes a zero-shot, training-free anomaly detection pipeline for industrial images that combines three foundation models: GPT-3 generates normal and anomaly text prompts, Grounding DINO localizes the product in the image, and CLIP scores the similarity between the cropped product image and the generated prompts. The method is evaluated on MVTec-AD and VisA, with reported AUROC/AUPR of 0.932/0.966 and 0.829/0.857, respectively, claimed to outperform the previous zero-shot baseline WinCLIP. The paper also reports ablations that remove the prompt generation and object detection components.

Significance. If the reported results are reproducible, the paper makes a useful practical contribution: it identifies two real limitations of prior CLIP-based zero-shot anomaly detection (fixed template prompts that are not class-specific, and whole-image or sliding-window features that ignore object scale) and proposes a concrete multimodal pipeline to address them. The idea of using LLM-generated class-specific prompts together with grounding-based cropping is sensible, and the improvements over WinCLIP on VisA in particular are encouraging. The paper's strengths are its clear problem framing, the use of two standard benchmarks, and straightforward ablations. However, the central prompt-generation step is described only abstractly, with no disclosure of the LLM inputs, output prompts, or sampling parameters, and the ablation discussion contains inconsistent attributions; these issues currently prevent the results from being independently verified.

major comments (5)
  1. [III-A, Eqs. (1)-(2)] The inputs x_normal and x_anomaly are never defined, the generated prompt sets P_normal and P_anomaly are never reported, and no GPT-3 sampling parameters (temperature, number of samples, seed) are given. Because the anomaly score in Eq. (4) is a softmax over CLIP similarities to exactly these generated prompts, the reported results in Tables II and III are conditioned on an unreported stochastic draw from a black-box language model. This makes the comparison to WinCLIP irreproducible and leaves open the possibility that the gains reflect one favorable prompt set rather than a robust property of the method.
  2. [IV-C, Tables IV-V] The ablation discussion attributes the wrong components. On MVTec-AD, removing object detection (0.932→0.913) is a 1.9pp drop, yet the text credits this to 'the language model-based prompt generation'; removing prompt generation (0.932→0.905) is a 2.7pp drop, yet the text credits it to 'the importance of object-level features'. On VisA the ordering of the two drops is reversed (3.7pp for object detection vs 1.4pp for prompt generation), so the textual attributions are inconsistent with the numbers in both datasets. These sentences must be rewritten so the explanation matches Tables IV and V.
  3. [IV-A] Experimental details are insufficient for reproduction. The CLIP backbone variant (e.g., RN50 vs ViT-B/32), the Grounding DINO version and detection threshold, the handling of multiple detected boxes, and the number of prompts per class are all unspecified. The absence of error bars or per-class AUROC/AUPR also makes the claimed margin over WinCLIP statistically unverifiable, since zero-shot prompt-based pipelines are sensitive to prompt selection and random sampling.
  4. [IV-B] The zero-shot comparison is incomplete. Among the baselines in Tables II and III, only WinCLIP is truly zero-shot; SPADE, PaDiM, and PatchCore are 1-shot methods. More recent zero-shot anomaly detection methods (e.g., AnomalyCLIP, WinCLIP+, APRIL-GAN) are not compared, so the claim of state-of-the-art zero-shot performance is not established by the presented evidence.
  5. [IV-B] The sentence 'outperforming the previous best zero-shot method, WinCLIP, by a significant margin of over 1 percentage point in both metrics' is contradicted by Table II: on MVTec-AD the AUPR gain over WinCLIP is 0.001 (0.966 vs 0.965), far below 1 percentage point. The claim should be stated separately per dataset and per metric.
minor comments (7)
  1. [Abstract] The abstract contains the fragment 'GPT-3. generate text prompts'; the missing 'to' should be inserted.
  2. [Introduction] The phrase 'efficient, accurate, and salable quality control' uses 'salable' where 'scalable' is intended.
  3. [IV-A] The word 'anootated' in the description of MVTec-AD should be 'annotated'.
  4. [References] The references section header appears twice ('REFERENCES REFERENCES'), which should be corrected.
  5. [V] The conclusion misdescribes the pipeline: it says the method 'first employs a deep learning-based anomaly detection model', then an LLM, then object detection, which contradicts the order in Section III (prompt generation, object localization, then CLIP anomaly scoring).
  6. [III-C, Eq. (4)] The anomaly score is defined as a ratio of raw dot products. CLIP embeddings are typically normalized to unit vectors, but the paper does not state this, and the ratio is undefined if the denominator approaches zero; please specify the exact similarity computation.
  7. [II-B] The related-work text says language models have been used for 'question answering, and sentiment analysis [5, 6]', but references [5] (WinCLIP) and [6] (GPT-3) are not sentiment-analysis papers; the citation is misplaced.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the method is an empirical zero-shot pipeline built from frozen external foundation models, and no reported result reduces to its own input or to a self-citation.

full rationale

The paper's derivation chain is a forward inference pipeline, not a fitted model: GPT-3 generates prompts, Grounding DINO crops objects, and CLIP scores the crops against the prompts. Equation (4) defines the anomaly score as a softmax ratio of CLIP similarities to the anomaly and normal prompt embeddings; this is the intended computation, not a self-definition, because the prompts are genuine inputs and the reported AUROC/AUPR values are measured against external dataset labels. I looked for the six circularity patterns and found none. There are no fitted parameters that are later renamed as predictions, no load-bearing self-citations, no imported uniqueness theorems, and no known result merely relabeled in new coordinates. The unreported GPT-3 inputs x_normal and x_anomaly, the unspecified prompt sets, and the stochasticity of a black-box language model are legitimate reproducibility and robustness concerns, and they could mean the headline numbers are not deterministic or not fully automatic; but they do not make the derivation circular, because no equation in the paper is equivalent to its own input by construction. The ablation tables further show that the two added components each contribute to performance, which is consistent with an empirical pipeline rather than a tautology. The central claim is a benchmark comparison, and the comparison is self-contained relative to the external models used; therefore the circularity score is 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the zero-shot generalization of three pretrained models, each treated as a black box. The paper introduces no new entities, but the LLM prompt design and detector thresholds are uncontrolled free parameters that directly affect the reported scores.

free parameters (3)
  • GPT-3 prompt inputs and generated prompt sets = not disclosed
    The exact x_normal and x_anomaly inputs and the resulting P_normal and P_anomaly text are not reported; they are effectively hand-designed via LLM prompting and directly determine CLIP similarity scores.
  • CLIP backbone variant = not reported
    The paper does not state whether CLIP ViT-B/32, ViT-L/14, or another variant is used; zero-shot anomaly detection performance depends heavily on this choice.
  • Object detection filtering and fusion parameters = not reported
    Grounding DINO may output multiple boxes; the paper does not specify confidence thresholds, box selection, or how multiple object patches are averaged into e_object.
assumptions (3)
  • ad hoc to paper GPT-3 can generate class-specific normal and anomaly prompts from only the product category name
    Invoked in Section III-A; no prompt examples or quality validation are provided, and the entire anomaly score depends on prompt quality.
  • domain assumption Grounding DINO can accurately localize the product in every test image given the class name
    Invoked in Section III-B; failure to detect or incorrect crops would corrupt the CLIP anomaly score, and detection accuracy is not reported.
  • domain assumption CLIP's text-image similarity provides a valid measure of normality and anomaly for industrial product images
    Invoked in Section III-C; the anomaly score is a softmax over CLIP similarities, which may not align with true defects.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Automatic Prompt Generation and Grounding Object Detection for Zero-Shot Image Anomaly Detection." pith.science (2026). https://pith.science/paper/QJRVE4DL

@misc{pith2026241119220,
  author       = {Pith},
  title        = {Pith review of: Automatic Prompt Generation and Grounding Object Detection for Zero-Shot Image Anomaly Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QJRVE4DL}},
  note         = {Machine review of arXiv:2411.19220}
}
read the original abstract

Identifying defects and anomalies in industrial products is a critical quality control task. Traditional manual inspection methods are slow, subjective, and error-prone. In this work, we propose a novel zero-shot training-free approach for automated industrial image anomaly detection using a multimodal machine learning pipeline, consisting of three foundation models. Our method first uses a large language model, i.e., GPT-3. generate text prompts describing the expected appearances of normal and abnormal products. We then use a grounding object detection model, called Grounding DINO, to locate the product in the image. Finally, we compare the cropped product image patches to the generated prompts using a zero-shot image-text matching model, called CLIP, to identify any anomalies. Our experiments on two datasets of industrial product images, namely MVTec-AD and VisA, demonstrate the effectiveness of this method, achieving high accuracy in detecting various types of defects and anomalies without the need for model training. Our proposed model enables efficient, scalable, and objective quality control in industrial manufacturing settings.

Figures

Figures reproduced from arXiv: 2411.19220 by the authors.

Figure 1
Figure 1. Illustration of the prompt generation and object localisation-enhanced [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

25 extracted references · 16 canonical work pages

  1. [1]

    Cut: A controllable, universal, and training-free visual anomaly generation framework,

    H. Sun, Y . Cao, and O. Fink, “Cut: A controllable, universal, and training-free visual anomaly generation framework,” arXiv preprint arXiv:2406.01078 , 2024

    arXiv 2024

  2. [2]

    A framework for industrial inspection system using deep learning,

    M. W. Hridoy, M. M. Rahman, and S. Sakib, “A framework for industrial inspection system using deep learning,” Annals of Data Science , vol. 11, no. 2, pp. 445–478, 2024

    work page 2024

  3. [3]

    Defect detection methods for industrial products using deep learning techniques: A review,

    A. Saberironaghi, J. Ren, and M. El-Gindy, “Defect detection methods for industrial products using deep learning techniques: A review,” Algorithms, vol. 16, no. 2, p. 95, 2023

    work page 2023

  4. [4]

    Learn- ing transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, et al. , “Learn- ing transferable visual models from natural language supervision,” in International conference on machine learning, PMLR, 2021, pp. 8748–8763

    work page 2021

  5. [5]

    Winclip: Zero-/few-shot anomaly classification and segmentation,

    J. Jeong, Y . Zou, T. Kim, D. Zhang, A. Ravichan- dran, and O. Dabeer, “Winclip: Zero-/few-shot anomaly classification and segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 19 606–19 616

    2023

  6. [6]

    Language models are few-shot learners,

    T. Brown, B. Mann, N. Ryder, et al., “Language models are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020

    work page 1901

  7. [7]

    Grounding dino: Marrying dino with grounded pre-training for open- set object detection,

    S. Liu, Z. Zeng, T. Ren, et al. , “Grounding dino: Marrying dino with grounded pre-training for open- set object detection,” arXiv preprint arXiv:2303.05499 , 2023

    arXiv 2023

  8. [8]

    Drocc: Deep robust one-class classification,

    S. Goyal, A. Raghunathan, M. Jain, H. V . Simhadri, and P. Jain, “Drocc: Deep robust one-class classification,” in International conference on machine learning , PMLR, 2020, pp. 3711–3721

    work page 2020

Show all 25 references
  1. [9]

    A spectrogram image-based network anomaly detection system using deep convolutional neural network,

    A. S. Khan, Z. Ahmad, J. Abdullah, and F. Ahmad, “A spectrogram image-based network anomaly detection system using deep convolutional neural network,” IEEE access, vol. 9, pp. 87 079–87 093, 2021

    2021

  2. [10]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, et al. , “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  3. [11]

    Visual instruc- tion tuning,

    H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruc- tion tuning,” Advances in neural information processing systems, vol. 36, 2024

    2024

  4. [12]

    Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face,

    Y . Shen, K. Song, X. Tan, D. Li, W. Lu, and Y . Zhuang, “Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face,” Advances in Neural Information Processing Systems , vol. 36, 2024

    2024

  5. [13]

    Bounding the capabilities of large language models in open text generation with prompt constraints,

    A. Lu, H. Zhang, Y . Zhang, X. Wang, and D. Yang, “Bounding the capabilities of large language models in open text generation with prompt constraints,” arXiv preprint arXiv:2302.09185, 2023

    2023 arXiv

  6. [14]

    Unsupervised prompt tuning for text-driven object detection,

    W. He, W. Chen, B. Chen, et al., “Unsupervised prompt tuning for text-driven object detection,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 2651–2661

    2023

  7. [15]

    Fast r-cnn,

    R. Girshick, “Fast r-cnn,” in Proceedings of the IEEE international conference on computer vision , 2015, pp. 1440–1448

    2015

  8. [16]

    You only look once: Unified, real-time object detec- tion,

    J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detec- tion,” in Proceedings of the IEEE conference on com- puter vision and pattern recognition, 2016, pp. 779–788

    2016

  9. [17]

    Weakly supervised object localization and detection: A survey,

    D. Zhang, J. Han, G. Cheng, and M.-H. Yang, “Weakly supervised object localization and detection: A survey,” IEEE transactions on pattern analysis and machine intelligence, vol. 44, no. 9, pp. 5866–5885, 2021

    2021

  10. [18]

    Zero-shot ground- ing of objects from natural language queries,

    A. Sadhu, K. Chen, and R. Nevatia, “Zero-shot ground- ing of objects from natural language queries,” in Pro- 5 ceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 4694–4703

    2019

  11. [19]

    Referring to objects in photographs of natural scenes,

    S. KAZEMZADE, V . Ordonez, M. MATTENV, et al. , “Referring to objects in photographs of natural scenes,” in Empirical Methods in Natural Language Processing , vol. 28, 2014, pp. 787–789

    2014

  12. [20]

    Clip-vg: Self-paced curriculum adapting of clip for visual grounding,

    L. Xiao, X. Yang, F. Peng, M. Yan, Y . Wang, and C. Xu, “Clip-vg: Self-paced curriculum adapting of clip for visual grounding,” IEEE Transactions on Multimedia , 2023

    2023

  13. [21]

    Mvtec ad–a comprehensive real-world dataset for un- supervised anomaly detection,

    P. Bergmann, M. Fauser, D. Sattlegger, and C. Steger, “Mvtec ad–a comprehensive real-world dataset for un- supervised anomaly detection,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 9592–9600

    2019

  14. [22]

    Spot-the-difference self-supervised pre-training for anomaly detection and segmentation,

    Y . Zou, J. Jeong, L. Pemula, D. Zhang, and O. Dabeer, “Spot-the-difference self-supervised pre-training for anomaly detection and segmentation,” in European Con- ference on Computer Vision , Springer, 2022, pp. 392– 408

    2022

  15. [23]

    Sub-image anomaly de- tection with deep pyramid correspondences,

    N. Cohen and Y . Hoshen, “Sub-image anomaly de- tection with deep pyramid correspondences,” arXiv preprint arXiv:2005.02357, 2020

    2005 arXiv

  16. [24]

    Padim: A patch distribution modeling framework for anomaly detection and localization,

    T. Defard, A. Setkov, A. Loesch, and R. Audigier, “Padim: A patch distribution modeling framework for anomaly detection and localization,” in International Conference on Pattern Recognition , Springer, 2021, pp. 475–489

    2021

  17. [25]

    Towards total recall in industrial anomaly detection,

    K. Roth, L. Pemula, J. Zepeda, B. Sch ¨olkopf, T. Brox, and P. Gehler, “Towards total recall in industrial anomaly detection,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2022, pp. 14 318–14 328. 6

    2022

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.