{"id":"fbc5812e-cd44-4bad-a235-fe0a403aecd7","arxiv_id":"2505.02626","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"VELM combines a fast anomaly detector with a prompted multimodal LLM to classify industrial defects, and introduces corrected anomaly-class labels for MVTec-AD and VisA.","lead":"VELM is a two-stage pipeline that uses a fast industrial anomaly detector to flag suspicious images, then a multimodal language model to classify the defect type from the image, a red contour, and a text prompt. The authors also release cleaned versions of two standard datasets, MVTec-AC and VisA-AC, and report 84% classification accuracy on MVTec-AC.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported 'accuracy' cannot be reconciled with Eq. (1): the formula defines mean per-class Jaccard/IoU, which would be far below the 84.0/81.4 tables unless the actual computed metric is something else. The headline numbers are therefore not interpretable until the metric is clarified or recomputed.","rationale":"The strongest claim is SOTA anomaly-classification accuracy. Before asking whether the new benchmark labels are correct, one must ask whether 'accuracy' is well defined. Eq. (1) is not macro accuracy but mean per-class Jaccard/IoU. In a one-label-per-image setting, each class term's denominator is the size of the union of true and predicted instances of that class, so true negatives are excluded. For a balanced K-class object with correct rate p, the Eq. (1) value is roughly p/K rather than p; the reported 84.0 on MVTec-AC and 81.4 on MVTec-AD are therefore inconsistent with the stated formula. This is observable from the manuscript alone and does not depend on external label audits. The reader noticed the metric oddity but selected label provenance as the weakest assumption; I would prioritize the metric issue because it undermines the numbers even under perfectly correct labels. The label concern is still valid: 36 corrected labels, merged classes, removed classes, no inter-annotator agreement, and no public label files in the paper. However, it is secondary to the internal inconsistency and should be checked after the metric is clarified. I would keep the reader's conditional verdict: the paper is a plausible engineering contribution with useful ablations and oracle experiments, but the headline numeric claim must be recomputed and reported under a clearly stated metric before it can be accepted. The abstract/body discrepancy (80.4 vs 81.4) further supports the need for this re-audit.","tokens_in":11609,"tokens_out":9547,"duration_ms":118457,"concrete_test":"Request or reconstruct the per-image predictions and ground-truth labels underlying Table 2 (DDAD+GPT-4o on MVTec-AC), then compute three quantities from the confusion matrix: (i) overall accuracy, (ii) macro recall = mean_c TP_c/(TP_c+FN_c), and (iii) Eq. (1) exactly as written = mean_c TP_c/(TP_c+FP_c+FN_c). Compare with the reported 84.0. If (iii) is near 0.42 and (i) or (ii) is near 0.84, the metric definition needs correction; if none matches, the numbers need re-auditing. Repeat the same three computations for Table 1's 81.4 on MVTec-AD. This one recomputation settles whether the reported 'accuracy' is defined as claimed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1 defines macro accuracy in Eq. (1) as (1/|O|) sum_o (1/|C_o|) sum_c TP_c/(TP_c+FP_c+FN_c). In a one-label-per-image classification setting, TP_c+FP_c+FN_c counts exactly the images that are truly class c or predicted as class c, excluding true negatives. This is the mean per-class Jaccard index, not accuracy. For a balanced three-class object at 84% per-class recall, each class has TP ≈ 0.28N and denominator ≈ 0.67N, so the Eq. (1) value would be roughly 0.42, not 0.84. The tables (e.g., Table 2, DDAD+GPT-4o mean 84.0) therefore cannot be produced by the published formula. Either the authors computed macro recall or overall accuracy and miswrote Eq. (1), or the reported numbers are not reproducible from the stated protocol. Because every numerical claim in the abstract and Tables 1-3 depends on this quantity, the central SOTA claim is currently uninterpretable. This concern is more immediate than the label-provenance issue: even with perfect ground truth, the reported metric does not match its definition.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes VELM, a two-stage pipeline for industrial anomaly classification that combines a fast unsupervised anomaly detector (DDAD or PatchCore) with a multimodal LLM (GPT-4o or GPT-4o-mini). When the detector flags an anomaly, the LLM receives a normal reference image, the query image, a red-contour visual prompt derived from the detector's localization, and a structured text prompt that defines the anomaly classes and a classification strategy. The authors also introduce two refined datasets, MVTec-AC and VisA-AC, with corrected, merged, and filtered anomaly class labels, and evaluate VELM on these and on the original MVTec-AD. They report state-of-the-art accuracy on MVTec-AD (81.4% in the body, 80.4% in the abstract) and strong performance on MVTec-AC (84.0% with DDAD+GPT-4o), together with ablations of the prompt components and a simulated anomaly-vs-defect experiment.","tokens_in":11774,"tokens_out":7464,"duration_ms":80472,"significance":"The two-stage design is practically motivated: using a fast detector to filter normal images and to localize anomalies before invoking the LLM is a sensible way to balance speed and semantic flexibility. The ablations in Table 5 clearly show the value of the reference image, visual prompt, and text descriptions. The paper provides a code link and proposes concrete benchmark refinements that could be useful to the community. However, the central quantitative claims depend on an apparently mis-defined evaluation metric and on benchmarks whose labels are authored by the same researchers without external validation or a public data release. Until these issues are resolved, the reported accuracy numbers and the claimed state of the art are not interpretable.","major_comments":[{"comment":"The 'macro accuracy' defined in Eq. (1) is not an accuracy measure; it is the mean per-class Jaccard index (intersection over union), because the denominator TP_c + FP_c + FN_c is the union of ground-truth and predicted positive images for class c and excludes true negatives. In a single-label multi-class setting, this quantity is far lower than conventional accuracy for the same predictions; for a balanced three-class problem with 84% per-class recall and precision, Eq. (1) would give about 0.72, and with realistic zero-shot precision it would be substantially lower. The values reported in Tables 2 and 3 (e.g., 84.0 and 87.6) are therefore not reproducible from the published formula. Please clarify the actual computed metric (e.g., macro recall or overall accuracy), correct the equation, or recompute every headline number. Since the abstract and Tables 1–3 all depend on this quantity, the central SOTA claim is currently uninterpretable.","section":"§4.1, Eq. (1)"},{"comment":"The MVTec-AC and VisA-AC benchmarks are central contributions, but their labels are the authors' manual corrections (36 corrected samples, merged classes, removed low-support classes) with no inter-annotator agreement, no external audit, and no public dataset link; the paper only provides a code repository. Moreover, the text prompts used for classification (Figure 4) encode the same class definitions and classification strategy that the authors used to assign the corrected labels. This creates a circularity risk: the LLM may be following the label-generation rules rather than independently recognizing visual anomalous patterns. Please release the datasets and a detailed annotation protocol, report inter-annotator statistics, and discuss how the evaluation avoids this circularity, for example by holding out a test set labeled by independent experts.","section":"§3.3, Tables 2–3, Figure 4"},{"comment":"The anomaly-vs-defect experiment does not provide a valid measure of the claimed capability. The 'Normal' column is trivially 1.0 for all but one object category because normal images are filtered before the LLM; the random designation of 30% of anomaly classes as 'negligible' conflates class-level severity with instance-level acceptability; and the 89.8% mean accuracy aggregates this trivial component with the two-class anomaly/defect split. Please redesign the experiment with instance-level severity labels and a proper three-way evaluation, or clearly restrict the claim to the distinction between the two anomaly groups.","section":"§4.3, Table 4"}],"minor_comments":[{"comment":"The reported accuracy on MVTec-AD is 80.4% in the abstract and 81.4% in Section 4.2.1 and Table 1; please correct the inconsistency and verify the margin over Echo, since 81.4 − 72.9 = 8.5, not 9.5.","section":"Abstract vs. §4.2.1, Table 1"},{"comment":"The paper gives no public link for the MVTec-AC and VisA-AC datasets, only the code repository; please provide a dataset release to enable reproducibility and independent verification.","section":"§3.3"},{"comment":"The evaluation randomly selects a reference normal image from the training set, but the random seed is not reported; since the results may depend on this choice, please report seeds or average over multiple selections.","section":"§4.2"},{"comment":"F1-score and Cohen's kappa are reported for VELM but not for the baseline methods Echo and MCAD; please add these metrics or state why they are unavailable.","section":"Table 1"},{"comment":"Please specify which four 'combined' anomaly classes were excluded and list the exact per-object class merges, so that the benchmark construction is fully reproducible.","section":"§3.3.1"},{"comment":"The phrase 'first anomaly classification framework' is too strong, as prior work (MCAD, Echo) already addresses anomaly classification; please soften this claim.","section":"§5"}],"recommendation":"major_revision","confidential_remarks":"The metric error in Eq. (1) is serious: all headline numbers appear to be computed with a different formula than the one published. Even if the authors correct the metric, the benchmarks are self-authored without external validation or public release, so I would advise the editor to require the dataset release and a description of the annotation reliability before the central claims can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about arXiv:2505.02626. First, the configuration is new and practically motivated: a fast anomaly detector filters normals, and a multimodal LLM classifies the flagged defects using red-contour visual prompts plus structured text prompts. That is a sensible plug-and-play design, and the ablation study gives it some support. Second, the headline numbers cannot be reconciled with the paper's stated metric, and that is a load-bearing problem.\n\nWhat is actually new here: pairing DDAD (or PatchCore) with GPT-4o for anomaly classification in a two-stage, training-free pipeline; the red-contour visual prompt borrowed from medical VLM work; and the MVTec-AC / VisA-AC label refinements, which correct mislabels, merge overlapping classes, and drop tiny ones. The ablations show the visual prompt and the anomaly descriptions both matter. If the evaluation were clean, this would be a useful contribution to industrial inspection.\n\nThe soft spots are serious. Section 4.1 calls Eq. (1) macro accuracy, but the formula is the mean per-class Jaccard index (TP over TP+FP+FN). In a one-label-per-image setting that cannot produce the 84.0 or 81.4 numbers in the tables. Either they computed macro recall or plain accuracy and miswrote the equation, or the numbers are not reproducible. Every headline claim depends on this quantity, so the central result is currently uninterpretable. There is also a minor internal inconsistency: the abstract says 80.4% on MVTec-AD, the body says 81.4%. The benchmark labels are self-authored, with no inter-annotator agreement and no public data link in the paper, and the prompts encode the same class definitions that produced the labels, which raises a real circularity concern. No external baseline is run on the new benchmarks, and the random reference-image selection means no variance is reported.\n\nThese are all fixable: publish the label files and full prompts, run MCAD and Echo on MVTec-AC/VisA-AC, report variance over reference images, and correct the metric definition. As it stands, I would not trust the numbers.\n\nWho should read this? The anomaly-detection and industrial-inspection crowd gets the most value, because the design pattern is transferable. I would send it to peer review—the idea deserves referee time—but with a request for major revision. The engineering is fine; the evaluation is not yet.","headline":"The pipeline idea is genuinely useful, but the headline accuracy numbers don't match the paper's own metric equation, so the SOTA claim is uninterpretable until they clarify what was computed.","tokens_in":12437,"tokens_out":1796,"would_cite":false,"duration_ms":23272,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A two-stage pipeline—fast anomaly detector plus multimodal LLM—classifies industrial defects at 84 percent accuracy without task-specific training.","keywords":["anomaly classification","industrial inspection","multimodal large language models","anomaly detection","MVTec-AD","VisA","vision-language models","zero-shot classification"],"falsifier":"Have independent annotators re-label a random subset of MVTec-AC and VisA-AC from the original images using the paper's class definitions and compare their labels with the published ones; low inter-annotator agreement or systematic disagreement with the original dataset labels would mean the reported accuracy figures are not a valid measure of classification performance.","tokens_in":11280,"feed_emoji":"🏭","tokens_out":7869,"duration_ms":84501,"temperature":0.7,"pith_summary":"The paper sets out to turn industrial anomaly detection into anomaly classification: instead of only flagging an image as normal or abnormal, a system should name the type of defect and support decisions about whether it matters. The proposed pipeline, VELM, pairs a fast unsupervised anomaly detector with a multimodal LLM (a language model that also takes images as input), and the LLM has no task-specific training. The detector filters normal images and outlines suspicious regions; the LLM, guided by a text prompt that defines the normal object, the candidate defect classes, and a decision strategy, assigns the class. On the original MVTec-AD benchmark the method reports 81.4 percent accuracy, five points above the previous best, and on the authors' relabeled MVTec-AC dataset it reaches 84.0 percent with a real detector. The paper also contributes corrected class labels for two standard datasets, arguing that reliable evaluation of classification requires cleaner annotations than existing detection benchmarks provide.","feed_headline":"A fast detector plus a language model sorts defects at 84%","feed_subtitle":"No task-specific training: a vision expert filters normal parts, then the LLM names the defect.","key_machinery":"The load-bearing mechanism is the division of labor between a fast pixel-level anomaly detector and a multimodal LLM, joined by two kinds of prompts. The visual prompt is a red-line contour overlay on the localized anomaly, presented next to the query image and a normal reference image; the text prompt is structured into a normal-object description, anomaly class definitions, and an explicit classification strategy. The detector's job is to keep classification cheap and accurate by filtering normal samples and telling the LLM where to look, while the LLM's job is to apply semantic knowledge and follow user-defined class definitions. Ablation results show that removing the anomaly descriptions causes the largest accuracy drop, and that both the reference image and the contour overlay contribute; the complete prompt gives the best accuracy, supporting the claim that both visual and textual context matter.","core_discovery":"The central claim is that a specialized localization module and a semantically rich language model can be composed into a classifier that needs no training for the classification task. In VELM, a pixel-level anomaly detector acts as the Vision Expert: it decides whether an image is normal, and when it detects an anomaly it produces an anomaly map. That map is converted into a visual prompt by drawing a red contour around the detected region, and the LLM receives the query image, the contour image, a normal reference image, and a structured text prompt containing a normal-object description, definitions of every anomaly class, and a classification strategy. The paper reports that on MVTec-AD this composition reaches 81.4 percent accuracy, outperforming prior anomaly classification methods by 5 percentage points; on the refined MVTec-AC benchmark it reaches 84.0 percent with the DDAD detector and 87.8 percent when ground-truth masks are used instead. The authors further claim the same pipeline separates benign anomalies from critical defects with 89.8 percent accuracy in a simulated three-way normal/anomaly/defect task.","pith_inferences":["One implication left implicit is that the random choice of a normal reference image introduces variance in the reported numbers; a natural extension is to evaluate across multiple reference images per query or to learn which reference best supports the prompt.","If the dependency on localization quality holds generally, then the practical bottleneck for new industrial objects is the detector, not the language model; swapping in a detector specialized per object family should be tested as a way to close the gap to the oracle result.","The closed-set limitation named in the paper suggests a next step: using the LLM's open vocabulary to output a description when the anomaly fits no user-defined class, which would measure how much the semantic model adds beyond the predefined labels."],"forward_implications":["Anomaly classification becomes a zero-shot task: with an off-the-shelf detector and a pretrained multimodal LLM, an inspection system can name defect types without collecting labeled defect examples for each new object.","Classification accuracy is bounded by localization quality; the gap between oracle masks (87.8 percent) and real detectors (84.0 percent for DDAD, 78.1 percent for PatchCore) implies that better anomaly segmentation directly raises classification performance.","Prompt structure is a real component of the method; removing anomaly descriptions lowers accuracy by about five points, so the way classes are defined in language is part of the approach, not a detail.","A single pipeline can triage anomalies into normal, negligible anomaly, and critical defect at 89.8 percent accuracy in the paper's simulated setting, which is the decision problem industrial inspection actually faces.","MVTec-AC and VisA-AC offer a benchmark for classification rather than detection, and their corrected and merged labels can support comparable evaluation of future methods."],"supporting_citations":[{"why":"It supplies the diffusion-based Vision Expert that localizes anomalies and filters normal images in the main pipeline.","marker":"[20]"},{"why":"It provides the original MVTec-AD dataset whose class labels the paper corrects and merges to form MVTec-AC.","marker":"[4]"},{"why":"It provides the VisA dataset whose labels are refined into VisA-AC.","marker":"[28]"},{"why":"It supplies the strongest prior baseline on MVTec-AD whose accuracy the paper claims to beat by five points.","marker":"[19]"},{"why":"It supplies the Echo baseline, another LLM-based anomaly understanding method, used for comparison on MVTec-AD.","marker":"[7]"},{"why":"It supplies PatchCore, the alternative vision expert used in ablations to show sensitivity to detector quality.","marker":"[23]"},{"why":"It motivates the red-line contour visual prompt that overlays localized anomalies for the multimodal LLM.","marker":"[11]"}],"fun_headline_variants":["No-training LLM pipeline hits 84% on industrial defect classification","Vision expert + LLM: anomaly classifier with zero task-specific training","LLM classifies defects without any training, 84% accuracy","Zero-shot defect classification via detector-prompted LLM"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline accuracy numbers rest on the paper's own manual relabeling of MVTec-AD and VisA—36 corrected samples, merged classes, and removal of small classes—being the correct ground truth, and no independent annotator or external audit checks those labels.","fun_headline_variants_meta":{"raw":{"variants":["No-training LLM pipeline hits 84% on industrial defect classification","Vision expert + LLM: anomaly classifier with zero task-specific training","LLM classifies defects without any training, 84% accuracy","Zero-shot defect classification via detector-prompted LLM"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000851,"raw_usage":{"total_tokens":3733,"prompt_tokens":1008,"completion_tokens":2725,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":624,"completion_tokens_details":{"reasoning_tokens":2652}},"tokens_in":624,"tokens_out":2725,"duration_ms":19996,"temperature":1.0,"reasoning_tokens":2652,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:46:44.297572+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have independent annotators re-label a random subset of MVTec-AC and VisA-AC from the original images using the paper's class definitions and compare their labels with the published ones; low inter-annotator agreement or systematic disagreement with the original dataset labels would mean the reported accuracy figures are not a valid measure of classification performance.","supporting_citations":[{"cited_title":"Mvtec ad–a comprehensive real-world dataset for unsupervised anomaly detection","cited_arxiv_id":null,"evidence_quote":"It provides the original MVTec-AD dataset whose class labels the paper corrects and merges to form MVTec-AC."},{"cited_title":"Spot-the-difference self-supervised pre- training for anomaly detection and segmentation","cited_arxiv_id":null,"evidence_quote":"It provides the VisA dataset whose labels are refined into VisA-AC."},{"cited_title":"Mcad: Multi- classification anomaly detection with relational knowledge distillation","cited_arxiv_id":null,"evidence_quote":"It supplies the strongest prior baseline on MVTec-AD whose accuracy the paper claims to beat by five points."},{"cited_title":"Towards to- tal recall in industrial anomaly detection","cited_arxiv_id":null,"evidence_quote":"It supplies PatchCore, the alternative vision expert used in ablations to show sensitivity to detector quality."}],"review_version":1}