{"id":"62513b4c-c585-4db7-91c2-8cc926c96896","arxiv_id":"2411.14515","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"MAD-Bench is a new benchmark for severity-aligned anomaly detection, and the paper finds MLLM-based scoring ranks severity better than conventional AD models.","lead":"The paper introduces a benchmark that tests whether anomaly detection models can rank image defects and medical findings by severity, not just normal versus abnormal. It reports that multimodal large language models align scores with severity better than standard anomaly detectors, though the comparison is not fully apples-to-apples.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MLLM prompts supply the same severity rubric that defines the ground-truth labels, confounding the central Finding 1 comparison.","rationale":"The paper contributes a useful benchmark and an extensive empirical study, and the reader's conditional verdict is appropriate. The reader's weakest assumption—manual severity labels—is legitimate, but the two externally labeled medical datasets (DRD-MAD, Covid19-MAD) provide some independent support for label validity, and MLLMs also outperform there. The more decisive and pervasive issue is that the MLLM prompts in Appendix B contain the very severity rubric used to build the ground-truth levels, on all six datasets, while conventional baselines are evaluated with no severity information. This makes Finding 1's interpretation—that prior knowledge and test-time references drive severity alignment—premature: the MLLMs may be following an answer key rather than exercising domain knowledge. A prompt-ablation experiment would separate these explanations cleanly. This does not overturn the reader's CONDITIONAL verdict but sharpens the required condition: the benchmark's headline comparison should be re-run with the rubric removed from the prompts before the superiority claim can be accepted.","tokens_in":35634,"tokens_out":7026,"duration_ms":69303,"concrete_test":"Run MMAD-4o and MMAD-Sonnet on MVTec-MAD, VisA-MAD, DRD-MAD, and Covid19-MAD using an ablated prompt that retains normal reference images and task context but removes all severity-level descriptions and score ranges (e.g., replace with 'assign an integer anomaly score from 0 to 100; higher means more severe'). Keep temperature=0 and all other settings identical. If the C-index on these datasets drops from the reported 77-96 range toward the conventional baselines' 60-80 range, Finding 1's MLLM advantage is largely rubric-driven; if performance remains stable, the domain-knowledge interpretation survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central comparison in Section 5.2, Finding 1, is confounded by an information asymmetry: the MLLM baselines (Section 4.3) receive, inside their prompts (Appendix B), the exact severity-level taxonomy and score ranges used to construct the ground-truth labels (Section 4.1). For MVTec-MAD and VisA-MAD, the prompt explicitly maps minor/moderate/severe defects to score intervals 1-30/31-70/71-100, mirroring the authors' Level 1/2/3 'minor/moderate/economically impactful/severe' rubric. Conventional baselines (Skip-GAN, PatchCore, RRD, etc.) receive no severity information at all; they simply output raw deviation scores. Thus, a high C-index for MLLMs may demonstrate only that a model handed the answer key can map images to it, not that domain knowledge or test-time references produce severity-aligned scores. The paper's zero-shot results (Table 6) remove normal references but still retain the textual severity rubric, so they do not isolate the rubric's effect. If the rubric were omitted from the prompts, the MLLMs' advantage might shrink or vanish. The finding is not internally inconsistent, but its interpretation as evidence for 'prior knowledge' is underdetermined by the reported experiments.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Multilevel Anomaly Detection (MAD), a setting in which an anomaly score should reflect the practical severity of an anomaly rather than only binary normality. It introduces MAD-Bench, a benchmark built from six datasets (MultiDogs-MAD, MVTec-MAD, VisA-MAD, DRD-MAD, Covid19-MAD, and SkinLesion-MAD) with multiple severity levels, and evaluates ten conventional unsupervised anomaly detectors alongside four MLLM-based baselines that receive a few normal reference images plus a textual prompt. Performance is measured by AUROC, C-index, and Kendall's Tau-b. The central finding is that MLLM-based models generally achieve better severity alignment than conventional models, which the authors attribute to prior knowledge and contextual reasoning. Six additional findings examine the correlation between binary and multilevel performance, bias toward anomalous area, detection per severity level, robustness when light anomalies are included as normal, and sensitivity to input corruption. The code and datasets are promised for public release.","tokens_in":35917,"tokens_out":4765,"duration_ms":44318,"significance":"The benchmark is a useful and timely resource: it defines a concrete evaluation protocol for severity-aware anomaly scores, covers multiple application domains, and includes both open-source conventional baselines and proprietary MLLM baselines with documented prompts. The emphasis on severity alignment addresses a real gap in anomaly detection evaluation. However, the headline claim that MLLMs are consistently better at severity alignment is undermined by an information asymmetry between the compared model families: the MLLM prompts in Appendix B contain the same severity taxonomy and score brackets that define the ground-truth labels in Section 4.1, while conventional models receive no semantic severity information. The manual severity labels for four datasets are also not externally validated. If these concerns are addressed, the benchmark could serve as a valuable community resource; in its current form, the central comparative finding should be regarded as not yet established.","major_comments":[{"comment":"All reported results are point estimates without error bars, confidence intervals, or significance tests. The averaged C-index values in Table 3 show small gaps between top-ranked models (e.g., MMAD-4o at 81.24 vs. MMAD-Sonnet at 79.77, and RRD at 76.14 vs. PNI at 74.37); without per-subset variance it is impossible to tell whether these differences, and the ranks derived from them, are meaningful. The paper's headline Finding 1 depends on such differences. I request that the authors report per-subset standard deviations or bootstrap intervals, and perform a paired significance test (e.g., Wilcoxon signed-rank across dataset subsets) for the MLLM-versus-conventional comparison.","section":"Tables 3–6; Appendix F"}],"minor_comments":[{"comment":"Reference [1] lists the author as 'Ahdasdwdasd,' which appears to be a placeholder or corrupted name; please verify the citation for the normal-skin dataset.","section":"References"},{"comment":"In the header of Table 10, the column 'Fryum' appears twice; the second occurrence is presumably a different subset and should be renamed or removed.","section":"Table 10"},{"comment":"The sentence 'For the remaining datasets, which contain class labels for each sample. (e.g., defect types, disease types, or class names), we manually assign severity levels based on these class labels.' contains a misplaced period after 'sample'; the sentence should be joined.","section":"Section 4.1"},{"comment":"The dataset name is spelled inconsistently: 'Visa-MAD' in Section 5.2, 'ViSA-MAD' in Figure 3, and 'VisA-MAD' elsewhere. Please standardize to 'VisA-MAD'.","section":"Throughout"},{"comment":"The prompts contain minor typos, including 'anwser' and 'even you are not sure about the anwser' in the MultiDogs prompt; please proofread the appendix text.","section":"Appendix B"},{"comment":"The header 'Leve 0' in Table 13 should be 'Level 0'.","section":"Table 13"},{"comment":"The table caption states that the best MLLM and best conventional baselines are highlighted in red and blue, but the visible reproduction contains no color; please ensure the colors are shown in the final version or use symbols.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The paper makes a strong claim about MLLMs outperforming conventional detectors in severity alignment, but the prompt-label circularity is a serious confound that the authors need to address with an ablation or a held-out label design. The manual severity labels also need external validation. The benchmark itself is a useful contribution and could be publishable after these revisions. I would also double-check reference [1], which appears corrupted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a serious attempt at a benchmark, but the central comparison is confounded. The MLLMs get the severity rubric in their prompts, while conventional baselines only get raw deviation scores; so Finding 1 (MLLMs better align with severity) is weaker than it looks. That said, the benchmark construction itself is useful: converting six datasets into severity-labeled MAD tasks, with C-index and Kendall's tau-b as evaluation, is a reasonable and needed contribution. The breadth of experiments (RQ1–RQ6) is real, and the conventional baseline results with detailed per-subset tables give the community a starting point.\n\nWhere I'd push back: the manual severity labels for MVTec, VisA, MultiDogs, and SkinLesion are assigned by the authors without inter-annotator agreement or external validation. If the rubric isn't what practitioners care about, the rankings measure the authors' rubric. The MLLM prompt issue is more than a detail: the prompts in Appendix B literally encode the same score intervals (e.g., 1–30, 31–70, 71–100) used to define the ground-truth levels, so part of the MLLM advantage could be text-conditioned mapping rather than visual reasoning. A cleaner comparison would give conventional baselines the same severity information or remove the rubric from the prompts. Also, all results are point estimates without variance, and code/data are promised but not released—for a benchmark paper, that's a real gap.\n\nNone of this is fatal; the setting is worth formalizing, and the empirical foundation is extensive. But the headline finding needs re-running with the rubric removed or controlled.\n\nRecommendation: send to peer review, with a request for release of artifacts, validation of the severity labels, and a deconfounded MLLM comparison. A serious referee can help turn this into a solid benchmark paper.","headline":"Useful benchmark idea and thorough experiments, but the headline MLLM advantage is confounded by the prompts giving away the severity rubric.","tokens_in":36440,"tokens_out":1449,"would_cite":false,"duration_ms":14513,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Anomaly scores that reflect severity: multimodal large language models outperform conventional detectors in a new six-dataset benchmark.","keywords":["multilevel anomaly detection","severity alignment","MAD-Bench","multimodal large language models","anomaly scores","C-index","Kendall Tau-b","benchmark"],"falsifier":"Re-annotate the severity levels of MVTec-MAD, VisA-MAD, MultiDogs-MAD, and SkinLesion-MAD independently by domain experts (e.g., plant engineers for MVTec, dermatologists for SkinLesion) and rerun the benchmark. If the relative ordering of MLLM versus conventional methods changes materially, the paper's central finding would collapse to an artifact of the authors' annotations.","tokens_in":1857,"feed_emoji":"🩺","tokens_out":1963,"duration_ms":39605,"temperature":0.7,"pith_summary":"This paper argues that standard anomaly detection scores measure deviation from normality, not the practical severity of an anomaly, and that this gap undermines real-world use. To make the gap measurable, the authors introduce Multilevel Anomaly Detection (MAD), a setting where the anomaly score should rank severity levels, and build MAD-Bench, a benchmark spanning one-class novelty detection, industrial inspection, and medical imaging. The central finding is that MLLM-based baselines, prompted with a few normal reference images, assign severity-aligned scores more consistently than conventional unsupervised anomaly detectors on all datasets except VisA-MAD. If the benchmark's severity labels are accepted, the paper establishes that prior knowledge and test-time reference comparison matter more for severity alignment than the inductive biases of reconstruction- or distance-based detectors.","feed_headline":"LLMs score anomaly severity better than classic detectors","feed_subtitle":"A new six-dataset benchmark shows multimodal models with reference images rank anomaly severity best, exposing blind spots in standard…","key_machinery":"The load-bearing object is the MAD benchmark itself: six datasets (MultiDogs-MAD, MVTec-MAD, VisA-MAD, DRD-MAD, Covid19-MAD, SkinLesion-MAD) with severity levels either taken from existing medical labels or manually assigned from class labels. The evaluation protocol uses two rank-based metrics: the C-index, a generalization of AUROC that measures whether higher-severity samples receive higher scores, and Kendall's Tau-b, a stricter measure that also penalizes score ties within the same severity level. The MLLM baselines work by a few-shot prompting scheme: three normal images plus one inference image are fed to the model with a domain-specific severity rubric, asking for a 0-100 score, so that the model's world knowledge and reasoning substitute for training on anomalous data.","core_discovery":"The paper's central claim, stated as Finding 1 in Section 5.2, is that MLLM-based models 'demonstrated consistently better performance than conventional methods across all datasets, except the VisA-MAD dataset,' when evaluated on severity alignment via C-index and Kendall's Tau-b. On MAD-Bench, the best MLLM baselines achieve an average C-index around 81%, versus roughly 74% for the best conventional methods, with the gap largest on medical datasets (DRD-MAD and Covid19-MAD), where conventional methods fall near 65%. The paper also reports three secondary findings: binary and multilevel detection metrics are strongly correlated overall, yet some models, especially MLLMs, rank better on multilevel than binary evaluation; conventional models are biased toward the spatial area of the anomalous region, while MLLMs show less of this bias; and all models degrade under input corruption, particularly when fine-grained features are required.","pith_inferences":["The paper's comparison is implicitly a test of whether semantic understanding can be separated from low-level feature deviation; if the finding holds, it suggests that severity is a semantic property that conventional detectors cannot capture without supervision.","A natural extension is a hybrid pipeline where a fast conventional detector localizes anomalies and an MLLM assigns the severity score; the paper itself flags this direction in Section 6, but does not test it.","Because the MLLM baselines were evaluated with temperature 0 and a fixed set of normal reference images, one could test how sensitive the rankings are to the choice of reference images, which the paper does not report.","The manual severity assignments for MVTec-MAD, VisA-MAD, MultiDogs-MAD, and SkinLesion-MAD depend on the authors' interpretation of economic or clinical impact; re-annotating the same data with domain experts could change the relative standing of the methods."],"forward_implications":["If severity alignment is the goal, conventional AD models cannot be evaluated by binary AUROC alone; MAD-Bench provides a protocol that any new detector can be measured against.","MLLM-based scoring, especially with few-shot normal references, becomes a strong baseline for any anomaly-detection application where severity ordering matters, such as medical triage or industrial quality control.","The documented area bias of conventional methods implies that detectors relying on spatial features will systematically under-score small but dangerous defects, which should be a known limitation in deployment.","The strong correlation between binary and multilevel metrics means that a model that fails at binary detection is unlikely to excel at severity ranking, simplifying model selection in practice.","The finding that all models lose performance under input corruption, particularly noise, suggests that robustness to image degradation must be tested before deploying severity-aware detectors in real environments."],"supporting_citations":[{"why":"Provides the MVTec dataset that is adapted into MVTec-MAD, establishing the industrial-inspection images and defect classes.","marker":"[5]"},{"why":"Provides the VisA dataset that is adapted into VisA-MAD, the one dataset where conventional methods beat the MLLM baselines.","marker":"[69]"},{"why":"PatchCore is one of the strongest conventional baselines, representing memory-bank methods in the severity-alignment comparison.","marker":"[39]"},{"why":"RD4AD is a knowledge-distillation baseline that achieves some of the best conventional C-index scores, anchoring the comparison against MLLMs.","marker":"[12]"},{"why":"RRD is another top conventional baseline and is used to rank conventional methods in the average performance table.","marker":"[49]"},{"why":"Supplies the diabetic retinopathy dataset with predefined 0-4 severity labels, allowing a test of MLLMs on expert-level medical severity.","marker":"[14]"},{"why":"Supplies the COVID-19 chest X-ray dataset with radiologist-annotated severity scores, a key medical benchmark.","marker":"[11]"},{"why":"Supplies the ISIC skin lesion images that are re-labeled into benign, precancerous, and cancerous severity levels for SkinLesion-MAD.","marker":"[10]"},{"why":"Defines the C-index metric that the paper generalizes to multilevel severity alignment, the main evaluation quantity.","marker":"[51]"},{"why":"Defines Kendall's Tau-b, the stricter severity-ranking metric used alongside the C-index.","marker":"[25]"}],"fun_headline_variants":["LLMs outperform classic detectors in anomaly severity ranking","New benchmark shows LLMs best at anomaly severity scoring","Severity alignment: LLMs lead anomaly detection benchmark","Multilevel AD: Multimodal LLMs top severity rankings","Benchmark reveals LLMs rank anomaly severity more accurately"],"cache_read_input_tokens":38528,"weakest_assumption_plain":"The benchmark's ground truth is the authors' manual severity assignments for four of the six datasets; if those assignments do not match the severity that practitioners actually care about, then every model ranking and finding measures the authors' rubric rather than practical severity.","fun_headline_variants_meta":{"raw":{"variants":["LLMs outperform classic detectors in anomaly severity ranking","New benchmark shows LLMs best at anomaly severity scoring","Severity alignment: LLMs lead anomaly detection benchmark","Multilevel AD: Multimodal LLMs top severity rankings","Benchmark reveals LLMs rank anomaly severity more accurately"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000772,"raw_usage":{"total_tokens":3431,"prompt_tokens":971,"completion_tokens":2460,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":587,"completion_tokens_details":{"reasoning_tokens":2383}},"tokens_in":587,"tokens_out":2460,"duration_ms":16867,"temperature":1.0,"reasoning_tokens":2383,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:28:23.656025+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate the severity levels of MVTec-MAD, VisA-MAD, MultiDogs-MAD, and SkinLesion-MAD independently by domain experts (e.g., plant engineers for MVTec, dermatologists for SkinLesion) and rerun the benchmark. If the relative ordering of MLLM versus conventional methods changes materially, the paper's central finding would collapse to an artifact of the authors' annotations.","supporting_citations":[{"cited_title":"Mvtec ad–a comprehensive real-world dataset for unsupervised anomaly detection","cited_arxiv_id":null,"evidence_quote":"Provides the MVTec dataset that is adapted into MVTec-MAD, establishing the industrial-inspection images and defect classes."},{"cited_title":"Spot-the-difference self-supervised pre- training for anomaly detection and segmentation","cited_arxiv_id":null,"evidence_quote":"Provides the VisA dataset that is adapted into VisA-MAD, the one dataset where conventional methods beat the MLLM baselines."},{"cited_title":"Towards to- tal recall in industrial anomaly detection","cited_arxiv_id":null,"evidence_quote":"PatchCore is one of the strongest conventional baselines, representing memory-bank methods in the severity-alignment comparison."},{"cited_title":"Anomaly detection via reverse distillation from one-class embedding","cited_arxiv_id":null,"evidence_quote":"RD4AD is a knowledge-distillation baseline that achieves some of the best conventional C-index scores, anchoring the comparison against MLLMs."},{"cited_title":"Revisiting reverse distillation for anomaly detection","cited_arxiv_id":null,"evidence_quote":"RRD is another top conventional baseline and is used to rank conventional methods in the average performance table."},{"cited_title":"Di- abetic retinopathy detection","cited_arxiv_id":null,"evidence_quote":"Supplies the diabetic retinopathy dataset with predefined 0-4 severity labels, allowing a test of MLLMs on expert-level medical severity."},{"cited_title":"Dataset for covid-19 segmen- tation and severity scoring","cited_arxiv_id":null,"evidence_quote":"Supplies the COVID-19 chest X-ray dataset with radiologist-annotated severity scores, a key medical benchmark."},{"cited_title":"On the c-statistics for eval- uating overall adequacy of risk prediction procedures with censored survival data","cited_arxiv_id":null,"evidence_quote":"Defines the C-index metric that the paper generalizes to multilevel severity alignment, the main evaluation quantity."},{"cited_title":"A new measure of rank correlation","cited_arxiv_id":null,"evidence_quote":"Defines Kendall's Tau-b, the stricter severity-ranking metric used alongside the C-index."}],"review_version":1}