{"id":"cbf48382-fa03-4a99-8d39-221c78e2ecfd","arxiv_id":"2502.07601","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Anomaly-OV, trained on the new Anomaly-Instruct-125k dataset, improves zero-shot detection of image anomalies and their textual explanations over generalist MLLMs.","lead":"This paper introduces a large dataset and benchmark that teach and evaluate AI systems for spotting and explaining image defects, and a specialist vision-language model that beats general models such as GPT-4o on that task. The value is a reusable resource and a model that can act as a visual inspection assistant in industrial and medical settings.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Zero-shot claim hinges on an ambiguous Stage-1 protocol: 'corresponding datasets are replaced by VisA' is unclear about whether VisA is used in training, so Table 2's single 88.6 average may mix contaminated or per-dataset models.","rationale":"The reader's conditional verdict identifies both the zero-shot protocol ambiguity and the CLIP-feature discriminative assumption as risks. I find the protocol ambiguity more load-bearing because it attacks the definition of the contribution itself. The LTFM feature assumption is an inductive bias that is empirically supported by significance-map visualizations and ablations; a failure there would degrade performance but would not negate the zero-shot framing. In contrast, if evaluation data appear in training, the reported gains over GPT-4o are not gains in the setting advertised, and the dataset/benchmark contribution is also tainted. The paper's own sentence in Section 5.1 creates the ambiguity, and the absence of released artifacts prevents a reader from resolving it. A single retrained-excluded evaluation is the minimal check. I therefore keep the reader's CONDITIONAL verdict unchanged, with the protocol clarification and artifact release as explicit conditions.","tokens_in":23365,"tokens_out":6815,"duration_ms":60579,"concrete_test":"Ask the authors to provide the exact Stage-1 data provenance and checkpoint provenance: (1) a complete list of which evaluation datasets (MVTec, VisA, AITEX, ELPV, BTAD, MPDD, BrainMRI, HeadCT, Br35H) appear in Anomaly-Instruct-125k or WebAD, including any images, crops, descriptions, or GPT-4o-generated content; (2) retrain the Stage-1 expert from scratch with every target dataset excluded and with the released WebAD snapshot, freeze it, and evaluate this single expert on all nine benchmarks. If 88.6 average AUROC and the Table 4 text-detection results are reproduced on held-out data, the concern is resolved. If the numbers require a different expert per benchmark or use VisA data to train the model evaluated on VisA, the 'zero-shot' claim should be relabeled and the headline comparisons re-run.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline claim is that Anomaly-OV achieves significant zero-shot detection and reasoning. For Table 2's 88.6 AUROC to be zero-shot, none of the nine evaluation datasets may appear in Stage-1 or Stage-2 training. Section 5.1 states that Stage 1 'utilizes all of the data with anomaly labels in Anomaly-Instruct-125k', and Anomaly-Instruct-125k is built from MVTec AD, BMAD, Anomaly-ShapeNet, Real3D-AD, MVTec-3D AD, plus WebAD. MVTec AD is therefore in the training set while also being a Table 2 benchmark. The next sentence, 'Similar to previous works [6,110], when evaluating the model on the datasets contained in the training set, the corresponding datasets are replaced by VisA', admits that a replacement procedure is needed, but does not say who replaces what. If MVTec is swapped out and VisA is swapped in for training, then VisA is no longer unseen and the VisA row (91.1) is not zero-shot. If MVTec requires a separately retrained expert, then Table 2's single 'Ours' row blends models with different training data, and the average 88.6 is not a single zero-shot model. The paper also does not report error bars or release code/data, so the sensitivity of the comparison cannot be assessed. This is not an accusation of leakage; it is a request for a precisely specified protocol, because the central claim is only meaningful under one reading.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Anomaly-Instruct-125k, a visual instruction tuning dataset for anomaly detection and reasoning, and VisA-D&R, a benchmark with human-reviewed labels for evaluating both detection and reasoning. It then proposes Anomaly-OV, which augments LLaVA-OneVision with an 'anomaly expert' trained in two stages: the expert uses Look-Twice Feature Matching (LTFM) to compute per-token anomaly significance maps and a Visual Token Selector to emphasize suspicious tokens for the LLM. The central empirical claims are that the anomaly expert reaches an average AUROC of 88.6 across nine zero-shot anomaly detection benchmarks (Table 2) and that Anomaly-OV outperforms generalist MLLMs such as GPT-4o on text-based detection and reasoning on VisA-D&R (Table 4). The paper also provides ablations, qualitative comparisons, and extensions to 3D and medical domains.","tokens_in":23750,"tokens_out":8523,"duration_ms":72002,"significance":"If the claims hold, this is a valuable contribution: it provides the first large-scale instruction dataset and benchmark specifically for anomaly detection with reasoning, and it demonstrates a modular design in which a lightweight expert guides an MLLM's attention to anomalous tokens. The human-reviewed benchmark and the explicit failure-case analysis are strengths. The central ideas are plausible, and the qualitative examples show a clear qualitative gap in fine-grained anomaly description between Anomaly-OV and both GPT-4o and LLaVA-OneVision. However, the zero-shot protocol and the reasoning evaluation need to be tightened before the headline claims can be fully accepted.","major_comments":[{"comment":"The zero-shot protocol is underspecified at the point where the paper's headline claim is made. Stage 1 'utilizes all of the data with anomaly labels in Anomaly-Instruct-125k,' and Section 4.1 lists MVTec AD and the BMAD training set as components of Anomaly-Instruct-125k. The following sentence—'when evaluating the model on the datasets contained in the training set, the corresponding datasets are replaced by VisA'—does not say whether VisA is used as a training substitute (making the VisA row in Table 2 non-zero-shot) or as an evaluation substitute (making the MVTec row require a separately trained expert). Under either reading, the 88.6 average in Table 2 is not a single zero-shot model. Please specify the exact training/evaluation split for each of the nine rows and report the average over rows that are genuinely unseen in training.","section":"5.1, Table 2"},{"comment":"The reasoning evaluation has a circularity risk that should be addressed explicitly. The instruction data in Anomaly-Instruct-125k are generated by prompting GPT-4o with bounding-box hints (Section 4.1 and A1.1), and the primary reasoning metric, GPT-Score, uses GPT-4 as a judge (Section 4.2). A model fine-tuned to imitate GPT-4o's annotation style is therefore rewarded for stylistic similarity rather than for perceptual correctness. The non-GPT metrics (ROUGE-L, SBERT) are less affected and should be reported as the primary evidence for the reasoning claim, along with a blind human evaluation on a random subset or a judge from a different model family.","section":"4.1, 5.3"},{"comment":"No variance or significance information is reported for any experimental result. Several headline gaps are small (e.g., Table 4 F1: 0.83 for Anomaly-OV-7B vs 0.76 for InternVL-2-8B and 0.75 for Qwen2-VL-7B), and Table 2 reports single AUROC values. Please report mean and standard deviation over at least three training/evaluation runs (or, for frozen commercial models, over multiple prompt/decoding seeds), and state the number of test images per benchmark so that the reader can judge whether the differences are meaningful.","section":"5.1, Tables 2 and 4"},{"comment":"The medical extension is at risk of training/test contamination. Anomaly-Instruct-125k includes the training set of BMAD (Section 4.1), and Stage 2 additionally uses LLaVA-Med data (A2). Table 6 evaluates Anomaly-OV on the BMAD testing set; the paper does not state whether any BMAD test images, or images of the same anatomy, were seen during instruction tuning. Please clarify the exact overlap and, if there is any, retrain without the BMAD training set or report the extension as a within-distribution fine-tuned result rather than evidence of zero-shot generalization.","section":"A2, Table 6"},{"comment":"The abstract and Section 5.3 describe Anomaly-OV as achieving 'significant improvements over advanced generalist models.' In Table 4, however, Anomaly-OV is fine-tuned on 125k anomaly instructions plus an additional 350k samples from LLaVA-OneVision and LLaVA-Med data, while the generalist baselines (GPT-4o, Qwen2-VL, InternVL-2, IXC-2.5) are used zero-shot. The only matched fine-tuned baseline is LLaVA-OV-0.5B*; there is no fine-tuned LLaVA-OV-7B*. The comparison therefore conflates specialist fine-tuning with the contribution of the LTFM token-selection module. Please add a matched 7B baseline fine-tuned on the same data, and rephrase the claim as improvement over zero-shot generalists.","section":"5.3, Table 4"}],"minor_comments":[{"comment":"The model name is inconsistent: 'LLaVA' appears as 'LLaV A' in Tables 3-5 and in the text; please use a consistent spelling.","section":"Throughout"},{"comment":"The caption says some results are borrowed from [6,110,114]; please include the exact evaluation protocols, dataset versions, and prompt templates used in those sources so the comparison is reproducible.","section":"Table 2"},{"comment":"Please specify the exact prompts used for text-based detection (e.g., whether the model is asked for a one-word 'Yes/No' answer) and state how partial or non-conforming answers are handled; the caption only says 'questions designed to elicit a one-word answer.'","section":"Section 5.3, Table 4"},{"comment":"The paper states that 10 VisA classes are selected for VisA-D&R but does not describe the selection criterion or what happens to the remaining 2 classes; this affects the benchmark's generalizability and should be stated explicitly.","section":"Section 4.2"},{"comment":"The supplementary says WebAD download links will be released, but the paper does not state whether Anomaly-Instruct-125k, VisA-D&R, evaluation code, and model weights will be released; since the contribution is partly a dataset and benchmark, a release plan is needed for reproducibility.","section":"Supplementary, A2"}],"recommendation":"major_revision","confidential_remarks":"The paper has substantial empirical contributions, but the zero-shot protocol ambiguity and the GPT-family evaluation loop are the core issues. If the authors can clarify the protocol, provide matched baselines and variance estimates, and state a release plan, the contribution is likely significant. I would not reject on these grounds; they are addressable in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know about this paper is that the dataset and benchmark are the real contribution, and the model is a plausible bonus. Anomaly-Instruct-125k and VisA-D&R fill a genuine gap: there wasn't a large instruction-tuning set or a reasoned-evaluation benchmark for anomaly detection before this. The human review of VisA-D&R labels is a plus, and the WebAD collection pipeline, though GPT-4o-dependent, is a sensible way to scale diversity. The LTFM mechanism is also a legitimate idea, not just a wrapper: selecting and amplifying suspicious tokens before feeding the LLM is the right inductive bias for this task, and the ablations support each component. The paper also shows honest failure cases and limitations, which I respect.\n\nThe soft spots are real, and the biggest one is the zero-shot protocol. Stage 1 trains on all anomaly-labeled data in Anomaly-Instruct-125k, which includes MVTec AD, and Table 2 evaluates on MVTec AD. The sentence \"the corresponding datasets are replaced by VisA\" is genuinely ambiguous: does it mean the model is retrained without MVTec, or that VisA substitutes for MVTec in the training set, or that the reported MVTec row comes from a different model? If it's the last one, the 88.6 average is not a single zero-shot model. This needs to be pinned down before the central claim is meaningful. The reasoning evaluation is also partially circular: GPT-4o generates the training data and GPT-4 serves as the judge for GPT-Score. That doesn't kill the detection result, which rests on AUROC and human-checked labels, but it does weaken the reasoning comparison.\n\nThere's no code, no data release, and no error bars, which is standard for the field but still frustrating for a paper whose main assets are data and a benchmark. The claims are likely partially true, and the qualitative examples show real gains over GPT-4o and LLaVA-OneVision, but the protocol ambiguity and missing artifacts prevent full verification.\n\nWho benefits: anyone working on ZSAD, MLLM-based inspection, or visual instruction tuning. The dataset and benchmark deserve to be used. My recommendation: send it to peer review, but make the zero-shot protocol explicit and release the artifacts. The paper is conditionally acceptable, not a rejection.","headline":"A genuinely useful dataset and benchmark wrapped around a plausible specialist MLLM, but the zero-shot protocol is ambiguous enough that the headline claim shouldn't be taken on faith.","tokens_in":24280,"tokens_out":1716,"would_cite":true,"duration_ms":17157,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Anomaly-OV uses a Look-Twice Feature Matching mechanism to make a multimodal LLM detect and explain fine-grained anomalies in classes never seen during training.","keywords":["zero-shot anomaly detection","multimodal large language models","anomaly reasoning","visual instruction tuning","Look-Twice Feature Matching","anomaly localization","VisA-D&R","Anomaly-Instruct-125k"],"falsifier":"Hold all VisA classes and all medical benchmarks out of Anomaly-Instruct-125k, train Anomaly-OV on the remaining data, and compare its significance maps with ground-truth anomaly masks on the held-out classes; if localization AUROC is near chance or text-based detection F1 drops to the level of GPT-4o, the claim that LTFM transfers zero-shot is falsified.","tokens_in":23192,"feed_emoji":"🔍","tokens_out":9058,"duration_ms":72787,"temperature":0.7,"pith_summary":"The paper sets out to show that zero-shot anomaly detection and reasoning can be combined in one visual assistant, and that the reason current multimodal LLMs fail at this task is that they cannot pick out the visual tokens that carry the defect. To that end it builds Anomaly-Instruct-125k, a 125k-sample visual instruction-tuning dataset covering industrial, medical, in-the-wild, and 3D images, and VisA-D&R, a human-reviewed benchmark of 1,761 industrial images for detection and reasoning. On that benchmark, Anomaly-OV-7B reaches 0.79 accuracy and 0.83 F1 for text-based anomaly detection, compared with GPT-4o's 0.70 and 0.68, and it scores higher on both low-level and complex reasoning metrics. Its anomaly expert alone reaches an average AUROC of 88.6 across nine zero-shot benchmarks. If the claim holds, inspection systems would no longer need per-class normal training data and could explain the cause and impact of each defect in plain language.","feed_headline":"Look twice: this model beats GPT-4o at finding and explaining defects","feed_subtitle":"An anomaly expert highlights suspicious image regions so language models can explain defects without per-class training.","key_machinery":"The load-bearing mechanism is Look-Twice Feature Matching (LTFM), a two-pass attention steered by an anomaly expert. In the look-back pass, the global feature $\\mathbf{v}^o_0$ of the original image is combined with two learnable embeddings, $e^+$ for anomalous patterns and $e^-$ for normal patterns, through level-specific linear layers and MLPs, producing anomaly and normality descriptions $d^+_i$ and $d^-_i$ for each of four ViT levels. In the forward pass, each patch feature $\\mathbf{v}^i_j$ is compared with both descriptions by cosine similarity and softmax to form a significance map $m^i_j$, and the four maps are averaged into $m_j$. The Visual Token Selector multiplies patch tokens by $m_j$, average-pools them to a $2\\times2$ grid, and uses a Q-Former with the original tokens as keys and values to output a compact set of emphasized tokens, plus a global anomaly vector $r(I)$ that yields an image-level score through a sigmoid MLP. These emphasized tokens are inserted into the LLM with an '<adv> suspicious feature:' prompt so the model attends to the selected regions. The mechanism embodies the paper's key claim: generalist MLLMs already encode anomaly information in their visual tokens but need an external expert to point at the right tokens.","core_discovery":"The central claim is that Anomaly-OneVision (Anomaly-OV), a specialist visual assistant built on LLaVA-OneVision, outperforms generalist multimodal LLMs at both zero-shot anomaly detection and anomaly reasoning. Its Look-Twice Feature Matching mechanism derives class-aware anomaly and normality descriptions from the image's own global feature vector plus two learnable prototypes $e^+$ and $e^-$, then scores every patch token at four ViT levels by cosine similarity to those descriptions to build significance maps. The Visual Token Selector emphasizes the suspicious tokens, spatially pools them, and aggregates them with the original tokens through a Q-Former; the selected tokens are presented to the LLM with an indication prompt whose strength depends on the predicted anomaly score. The authors report that this design lets Anomaly-OV find defects that GPT-4o and LLaVA-OneVision miss, and that it achieves the best zero-shot AUROC among compared methods while also improving low-level and complex reasoning on VisA-D&R.","pith_inferences":["The paper's reported failure cases suggest the remaining bottleneck is the anomaly expert rather than the LLM, so a stronger expert could yield further gains without retraining the language model.","Because GPT-4o detects tiny defects when given bounding-box hints, the same token-selection idea may transfer to other fine-grained localization tasks, such as medical imaging or document inspection, where generalist MLLMs underuse low-level visual tokens.","The WebAD automatic collection pipeline implies that large in-the-wild web corpora can substitute for scarce industrial anomaly data; a testable extension is scaling WebAD and checking whether zero-shot AUROC keeps improving.","Passing multi-level ViT features directly into the LLM, which the paper lists as future work, is a natural next step because anomaly cues are often low-level and current MLLMs receive only the last encoder layer."],"forward_implications":["Zero-shot anomaly detection works without a text encoder: the image's own global features plus two learned prototypes supply the anomaly and normality descriptions.","The dataset is reusable: fine-tuning LLaVA-OneVision-0.5B on Anomaly-Instruct-125k raises text-based detection accuracy from 0.54 to 0.71 and F1 from 0.28 to 0.76.","The anomaly expert transfers across industrial and medical domains, averaging 88.6 AUROC on nine zero-shot benchmarks with no per-class training.","A specialist token-selection module can be added to an existing MLLM without large-scale pretraining, and it preserves the base model's ability to follow ordinary image instructions.","The same architecture extends to multi-view 3D and medical image reasoning, pointing toward a single assistant for visual inspection."],"supporting_citations":[{"why":"Supplies the base MLLM (LLaVA-OneVision) that Anomaly-OV extends and the main open-sourced comparison model.","marker":"[44]"},{"why":"Provides the CLIP visual encoder and the cosine-similarity zero-shot mechanism that LTFM adapts.","marker":"[73]"},{"why":"Contributes the multi-level ViT feature extraction and the object-agnostic prompt-learning baseline that Anomaly-OV builds on.","marker":"[110]"},{"why":"Defines one of the main zero-shot anomaly detection methods and prompt-ensemble baselines that the anomaly expert must beat.","marker":"[38]"},{"why":"Supplies the Q-Former used in the Visual Token Selector to aggregate suspicious tokens.","marker":"[49]"},{"why":"GPT-4o is used to generate the instruction data and is the main generalist comparison for detection and reasoning.","marker":"[72]"},{"why":"Provides MVTec AD, an industrial dataset used in Anomaly-Instruct-125k and in the nine-benchmark zero-shot evaluation.","marker":"[2]"},{"why":"Provides the VisA dataset from which the VisA-D&R benchmark is constructed.","marker":"[115]"}],"fun_headline_variants":["Anomaly-OV beats GPT-4o at zero-shot anomaly detection","Look-twice feature matching helps MLLM explain defects better than GPT-4o","Specialist visual assistant outperforms GPT-4o in anomaly reasoning","Zero-shot anomaly spotter with a look-twice mechanism exceeds GPT-4o","Find and explain image defects without per-class training"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method depends on the assumption that cosine similarity between patch features and two learned anomaly/normal description vectors reliably marks the true defect in classes and domains never seen during training; if that similarity signal misleads, the significance maps point the language model at irrelevant regions and both detection and reasoning break down.","fun_headline_variants_meta":{"raw":{"variants":["Anomaly-OV beats GPT-4o at zero-shot anomaly detection","Look-twice feature matching helps MLLM explain defects better than GPT-4o","Specialist visual assistant outperforms GPT-4o in anomaly reasoning","Zero-shot anomaly spotter with a look-twice mechanism exceeds GPT-4o","Find and explain image defects without per-class training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000368,"raw_usage":{"total_tokens":2008,"prompt_tokens":1013,"completion_tokens":995,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":629,"completion_tokens_details":{"reasoning_tokens":899}},"tokens_in":629,"tokens_out":995,"duration_ms":9315,"temperature":1.0,"reasoning_tokens":899,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T12:10:24.762496+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hold all VisA classes and all medical benchmarks out of Anomaly-Instruct-125k, train Anomaly-OV on the remaining data, and compare its significance maps with ground-truth anomaly masks on the held-out classes; if localization AUROC is near chance or text-based detection F1 drops to the level of GPT-4o, the claim that LTFM transfers zero-shot is falsified.","supporting_citations":[{"cited_title":"Llava-onevision: Easy visual task transfer, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the base MLLM (LLaVA-OneVision) that Anomaly-OV extends and the main open-sourced comparison model."},{"cited_title":"Learn- ing transferable visual models from natural language super- vision","cited_arxiv_id":null,"evidence_quote":"Provides the CLIP visual encoder and the cosine-similarity zero-shot mechanism that LTFM adapts."},{"cited_title":"AnomalyCLIP: Object-agnostic prompt learning for zero-shot anomaly detection","cited_arxiv_id":null,"evidence_quote":"Contributes the multi-level ViT feature extraction and the object-agnostic prompt-learning baseline that Anomaly-OV builds on."},{"cited_title":"Blip- 2: Bootstrapping language-image pre-training with frozen image encoders and large language models","cited_arxiv_id":null,"evidence_quote":"Supplies the Q-Former used in the Visual Token Selector to aggregate suspicious tokens."},{"cited_title":"Gpt-4o system card, 2024","cited_arxiv_id":null,"evidence_quote":"GPT-4o is used to generate the instruction data and is the main generalist comparison for detection and reasoning."},{"cited_title":"A scratchedcar","cited_arxiv_id":null,"evidence_quote":"Provides the VisA dataset from which the VisA-D&R benchmark is constructed."}],"review_version":1}