{"id":"5b1d06e9-0739-4e39-9bad-186faac870aa","arxiv_id":"2411.09310","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"CLIP-based zero-shot anomaly detection models achieve 3D Dice scores below 0.35 on BraTS-MET brain metastasis segmentation, indicating they are not clinically ready.","lead":"This paper tests four CLIP-based computer vision models that claim to spot anomalies without prior training on the target disease, using them to find brain metastases in MRI scans. None performed well enough for clinical use, even after adapting them on medical data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported 'Dice scores below 50%' is contradicted by the same table's F1-max values (0.58 for AdaCLIP Brats, 0.74 for VAND PMC), and several AUROC values below 0.5 suggest score-direction issues, making the central claim threshold-dependent.","rationale":"The reader's weakest assumption concerns external representativeness (prompt choice, single MRI sequence, 2D stacking), which is a legitimate limitation acknowledged in the paper. I focus instead on an internal inconsistency that strikes at the reported evidence itself. Because the paper equates Dice with F1, the F1-max column in Table 1 represents the best achievable Dice at some threshold. With F1-max values of 0.58 (AdaCLIP Brats) and 0.74 (VAND PMC), the statement that all Dice scores are below 50% is not universally true even within the authors' own results. The presence of AUROC values below 0.5 for several configurations further suggests the anomaly-score polarity is wrong in those runs, meaning the reported Dices may not reflect how the scores would actually be used. This is more load-bearing than the representativeness concern because it questions the validity of the reported numbers rather than only their generalizability. A revision must specify the thresholding rule, justify the chosen operating point, and reconcile the F1-max results with the conclusion. The reader's CONDITIONAL verdict remains appropriate, so I recommend no change to the verdict.","tokens_in":8502,"tokens_out":12496,"duration_ms":122384,"concrete_test":"Recompute per-volume Dice for each model at the global threshold that maximizes F1 (the F1-max operating point), and for configurations with AUROC<0.5, also recompute after flipping the anomaly scores. If VAND PMC or AdaCLIP Brats per-volume Dice exceeds 0.5 at that operating point, the statement 'Dice below 50%' fails and the conclusion must be revised to describe threshold-dependent performance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Section 3.4 ('Dice scores below 50%') is not internally consistent with Table 1. The paper defines Dice as equivalent to F1 (Section 2.3), yet the F1-max column—which is the maximum F1/Dice over thresholds—contains values above 0.5: AdaCLIP Brats has F1-max=0.58 and VAND PMC has F1-max=0.74. Therefore these models can achieve Dice above 50% when the threshold is selected appropriately. The reported Dice column is evidently computed at a single, unspecified threshold, and the paper provides no clinical rationale for that operating point. Additionally, several configurations have AUROC below chance (VAND PMC 0.29, AdaCLIP Brats 0.35, VAND Brats 0.44, AnomalyCLIP Brats 0.43), indicating the anomaly scores are anti-correlated with the ground truth in those runs; this suggests a score-inversion or implementation mismatch that is never discussed. The blanket conclusion 'consistently low performance' is thus not robust to threshold choice and is contradicted by the paper's own metrics.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates four CLIP-based zero-shot anomaly detection methods (AnomalyCLIP, VAND, AnoVL, AdaCLIP) on brain metastasis segmentation using the BraTS-METS 2023 dataset, under four training setups (industrial-pretrained, BraTS-finetuned, BraTS-from-scratch, and PMC-CLIP-backbone). The authors report Dice scores below 0.5 for all methods and conclude that CLIP-based ZSAD models are not yet suitable for clinical use on this 3D MRI task. The evaluation uses only an internal 70/30 split of the BraTS training set, with no independent test cohort, and a subsampled training set for AdaCLIP. The paper includes per-subject slice-level analyses, spatial error distributions, and correlation analyses. I find the negative claim directionally plausible but not fully supported by the paper's own metrics, because the reported F1-max values exceed 0.5 for several configurations and multiple AUROC values fall below 0.5, suggesting score-direction problems that are not discussed.","tokens_in":8755,"tokens_out":3333,"duration_ms":31999,"significance":"If the central negative claim were rigorously established, the paper would provide a useful falsification data point for the medical ZSAD community: it benchmarks four published methods on a clinically relevant 3D MRI dataset using open-source implementations and reports multiple segmentation metrics. This is valuable because most ZSAD evaluations in medical imaging are limited to 2D slices and do not include a 3D volumetric assessment. However, the current presentation undermines the strength of the conclusion: the inconsistency between the reported Dice scores and the F1-max column, together with AUROC values below chance, means the paper does not yet convincingly demonstrate that 'CLIP-based models are not ready' across reasonable operating points. The significance is therefore contingent on reanalysis and clarification of the threshold and score-orientation issues.","major_comments":[{"comment":"The central claim that \"Dice scores below 50%\" for all models is not internally consistent with Table 1. Section 2.3 defines Dice as equivalent to F1, yet the F1-max column contains values above 0.5: AdaCLIP Brats has F1-max = 0.58 and VAND PMC has F1-max = 0.74. This means that at the operating point that maximizes F1/Dice, these models exceed the 50% threshold. The reported Dice column appears to be computed at a single, unspecified threshold; the paper gives no clinical or methodological rationale for that threshold. The blanket conclusion in Section 3.4 therefore overstates the evidence. Please specify the threshold used for the Dice column, report Dice at the F1-max operating point, show PR curves, or otherwise justify the chosen operating point, and revise the central claim accordingly.","section":"Section 3.4 and Table 1"},{"comment":"Several AUROC values are below 0.5 (VAND PMC 0.29, AdaCLIP Brats 0.35, AnomalyCLIP Finetune 0.42, AnomalyCLIP Brats 0.43, VAND Brats 0.44). An AUROC below 0.5 indicates that the anomaly scores are anti-correlated with the ground truth, which is a symptom of score inversion, label mismatch, or an implementation error rather than a benign performance level. Section 3.1 attributes only the discrepancy between AUROC and Dice to class imbalance, and does not address why several AUROC values fall below chance. This is a load-bearing issue for the benchmark's validity because it suggests that the models' anomaly maps may be systematically reversed in some configurations. Please investigate and report whether the scores were inadvertently inverted, whether the ground-truth mask convention was reversed, or whether genuine anti-correlation occurs, and re-evaluate the affected setups with corrected or explicitly oriented scores.","section":"Table 1"},{"comment":"The evaluation is performed on a single internal 70/30 split of the BraTS-METS training set, with no independent test set, no cross-validation, and no statistical significance tests. The test set contains about 50 patients, and the reported Dice scores have large standard deviations (e.g., 0.35 ± 0.30). Without confidence intervals for the means or paired tests across subjects, the claim that the performance is \"consistently low\" across models is not statistically substantiated. The negative conclusion would be considerably stronger if the authors reported per-subject confidence intervals, performed tests comparing methods or setups, and ideally validated on the official BraTS-METS validation or test data.","section":"Section 2.1"},{"comment":"AdaCLIP was trained on only 50 randomly sampled patients, while the other methods appear to have been trained on the full 70% training split (about 115 patients). This introduces a training-set size confound when comparing AdaCLIP's Brats and PMC results with the other methods, and the paper's conclusion treats all methods equally. Section 3.4 acknowledges the subsampling as a limitation, but the conclusion \"consistently low performance for all models\" is not justified for AdaCLIP, whose lower Dice may be partly due to the reduced training data. The authors should either train AdaCLIP on the same number of patients as the other methods, or clearly bracket any cross-method comparisons involving AdaCLIP.","section":"Section 2.3"}],"minor_comments":[{"comment":"The abstract says the methods operate \"with no or minimal supervision,\" but two of the four setups (Finetune and Brats) train the adapters on labeled BraTS data. Please clarify the supervision degree in the abstract or add a qualifying phrase.","section":"Abstract"},{"comment":"Typo: \"we used the the BraTS 2023 Brain Metastases\" should read \"we used the BraTS 2023 Brain Metastases dataset.\"","section":"Section 2.1"},{"comment":"The text uses corrupted formatting \"V ANDexhibits\" and \"V ANDand\"; these should be \"VAND exhibits\" and \"VAND.\"","section":"Section 3.2"},{"comment":"The caption calls the plots \"Boxplots,\" but the text describes bars for 2D Dice scores and red crosses for 3D Dice scores; the figure caption is inconsistent with the content.","section":"Figure 2"},{"comment":"The phrase \"Dice scores below 50%\" should specify whether this refers to the mean, median, or per-model scores and at which threshold. Without this detail, the sentence is ambiguous given the F1-max values.","section":"Section 3.4"},{"comment":"The table formatting is hard to parse: values for Prompting depth, Prompting type, and K-clusters appear to be missing or misaligned for some models. Please align the columns or use a table with explicit per-model entries.","section":"Appendix Table 2"}],"recommendation":"major_revision","confidential_remarks":"The reader's report and the stress-test note converge on the same core problem: the paper's headline negative claim is threshold-dependent and is contradicted by the F1-max values in its own Table 1. The AUROC values below 0.5 are a separate red flag that may indicate a reversible implementation issue; if the scores are indeed inverted for some setups, the quantitative results could change substantially. I recommend that the journal ask for a revision that (a) clarifies and justifies the threshold used for Dice, (b) reports results at the F1-max operating point, (c) investigates and corrects the score-direction issue, and (d) adds statistical support for the negative claim. The paper's topic is timely and the negative result is potentially useful, but the current evidence is not yet internally consistent."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this paper is a useful, honest negative result. CLIP-based zero-shot anomaly detection methods, as configured here, do not produce clinically usable brain metastasis segmentations on BraTS-MET. The authors test four published methods across four setups including a PMC-CLIP backbone, on 2D axial slices with 3D stacking, and report Dice, AUROC, F1-max, per-subject variability, and spatial error analysis. The contribution is a new benchmark rather than new methodology, and that is fine.\n\nWhat it does well: it uses open-source implementations, keeps default hyperparameters mostly, discloses the AdaCLIP subsampling and prompt limitations, and flags the AUROC-vs-Dice discrepancy—a real issue in imbalanced volumetric segmentation. The per-subject and region-wise analyses are a nice touch, showing where models fail spatially.\n\nSoft spots, in proportion. The stress-test note about F1-max is legitimate. Section 3.4 says “Dice scores below 50%”, but Table 1's F1-max values are above 0.5 for several setups (AdaCLIP Brats 0.58, VAND PMC 0.74). Since the paper defines Dice as equivalent to F1, the reported Dice is evidently computed at a fixed threshold, and the F1-max shows the models can do better with threshold selection. The paper never discusses threshold choice or why the fixed threshold is clinically relevant. Worse, four configurations have AUROC below 0.5 (VAND PMC 0.29, AdaCLIP Brats 0.35, VAND Brats 0.44, AnomalyCLIP Brats 0.43), which indicates the anomaly scores are anti-correlated with ground truth—a score-inversion issue that is never mentioned. That undercuts the “consistently low performance” language, though it does not overturn the overall conclusion: even the best F1-max values are modest for clinical use, and the benchmark is what it is.\n\nOther softer spots: no independent test set (70/30 internal split of the training set), no statistical significance tests, AdaCLIP trained on only 50 patients, and no strong non-CLIP baseline for comparison. The conclusion is thus specific to these configurations, not a general verdict on CLIP-based medical ZSAD; the authors largely acknowledge this.\n\nWho is it for: anyone working on medical anomaly detection or CLIP adaptation. It deserves a serious referee as a benchmark/negative-results paper, but the authors should reconcile the F1-max and sub-0.5 AUROC values and discuss the operating point. A solid, honest piece of work that needs tightening rather than rethinking.","headline":"Useful and honest CLIP-ZSAD benchmark on brain metastases, but the reported Dice/F1-max and sub-0.5 AUROC discrepancies need reconciliation before the negative result is fully trustworthy.","tokens_in":9268,"tokens_out":2306,"would_cite":true,"duration_ms":19193,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Four CLIP-based zero-shot anomaly detection methods fail to reach clinically usable segmentation quality on brain metastasis MRI, with all mean 3D Dice scores below 50% even after adaptation to the BraTS-MET training set.","keywords":["anomaly detection","zero-shot learning","CLIP","medical imaging","brain metastasis segmentation","BraTS-MET","domain generalization","vision-language models"],"falsifier":"If an independent evaluation with specialized metastasis prompts, a 3D-aware aggregation, or a different CLIP-like backbone produced a mean 3D Dice above 0.50 on the same BraTS-MET test split without training on BraTS labels, the claim that CLIP-based ZSAD is not ready would be falsified for that configuration; conversely, reproducing the below-0.50 result under those improved conditions would confirm the paper's conclusion.","tokens_in":8310,"feed_emoji":"🧠","tokens_out":7382,"duration_ms":98104,"temperature":0.7,"pith_summary":"This paper asks whether CLIP-based zero-shot anomaly detection (ZSAD), which performs well on industrial defect inspection, is ready for clinical brain imaging. The authors benchmark four CLIP-based methods—AnomalyCLIP, VAND, AnoVL, and AdaCLIP—on brain metastasis segmentation using the BraTS-MET dataset, under four setups ranging from industrial pretraining to training adapters on the medical data with a biomedical CLIP backbone. Every configuration yields mean 3D Dice scores below 50%, with the best result at 0.35, and the authors argue that high AUROC values in prior work overstate localization quality because metastases occupy very few voxels. The conclusion is that CLIP-based ZSAD is not yet clinically usable for brain metastasis detection and requires substantial domain-specific adaptation.","feed_headline":"Zero-shot CLIP models miss brain tumors in MRI scans","feed_subtitle":"Four CLIP-based detectors all score below 50% Dice on BraTS-MET brain metastasis segmentation.","key_machinery":"The shared mechanism is the patch-level cosine-similarity anomaly map: for each image patch, the model compares the patch embedding's similarity to a text prompt describing an anomalous state against a prompt describing a normal state, producing a per-pixel score $M_{i,j}$. The four evaluated methods are variants of this idea: AnomalyCLIP learns object-agnostic text prompts, VAND adds trainable linear layers and memory banks of reference images, AnoVL introduces value-wise attention and test-time adaptation, and AdaCLIP combines static and per-image dynamic prompts with a semantic fusion module. The benchmark stacks the per-slice maps into a 3D volume and evaluates them with Dice, sensitivity, specificity, PPV, AUROC, and F1-max, keeping default hyperparameters and only changing the input size to $240 \\times 240$.","core_discovery":"The central claim is a negative result: on axial T2-weighted slices from BraTS-MET, none of the four CLIP-based zero-shot anomaly detection methods reaches clinically acceptable segmentation quality. The best mean 3D Dice is 0.35 (AdaCLIP trained from scratch on BraTS), and switching the backbone to PMC-CLIP, a biomedical-pretrained vision-language model, does not fix the problem, with VAND reaching 0.32 and AnomalyCLIP 0.16. The paper further claims that AUROC is an inflated performance estimate for this setting because the anomaly class is extremely sparse, so pixel-level Dice or F1 should be the primary metric. In the authors' reading, the gap between industrial and medical performance shows that CLIP embeddings and the current prompt and architecture designs do not yet capture the variability of brain metastases.","pith_inferences":["A direct test of the paper's implicit diagnosis would be to optimize prompt templates and hyperparameters for brain metastasis MRI while keeping the same adapters; if Dice then crosses 0.5, the bottleneck is the transfer setup rather than CLIP's embeddings.","The near-universal pattern of low Dice with high sensitivity suggests these models may be detecting image-level distribution shift rather than localized lesions, which would make them unsuitable for screening even when image-level AUROC looks high.","The peripheral-brain drop in Dice points to a spatial bias that could be probed by reweighting slices or using attention at the volume level, an extension the paper does not test."],"forward_implications":["CLIP-based zero-shot anomaly detection should not be deployed for brain metastasis screening or segmentation until domain-specific adaptation closes the gap.","Benchmark reports for medical ZSAD should report pixel-level Dice or F1-max alongside AUROC, since AUROC alone can hide poor localization on sparse lesions.","Medical-domain pretraining, such as PMC-CLIP, is not a sufficient substitute for task-specific adaptation on complex 3D data.","The 2D-to-3D stacking strategy is a likely source of error; volume-aware models are a concrete design direction the paper identifies.","Prompt design is an under-explored variable; the paper notes its default prompts may have disadvantaged the methods."],"supporting_citations":[{"why":"Supplies the CLIP vision-language model and the zero-shot classification paradigm that all four evaluated methods build on.","marker":"[3]"},{"why":"Provides the BraTS-MET dataset and the whole-tumor ground-truth masks used for evaluation.","marker":"[5]"},{"why":"Supplies AnomalyCLIP, one of the four evaluated ZSAD methods, with object-agnostic learnable prompts.","marker":"[6]"},{"why":"Supplies VAND, the zero-shot method with trainable linear layers and memory banks.","marker":"[7]"},{"why":"Supplies AnoVL, the value-wise attention method with test-time adaptation.","marker":"[8]"},{"why":"Supplies AdaCLIP, the hybrid static-dynamic prompt method with semantic fusion.","marker":"[9]"},{"why":"Provides PMC-CLIP, the biomedical-pretrained backbone used as an alternative to the original CLIP.","marker":"[10]"},{"why":"Supplies the F1-max metric and the aggregation convention used for comparing anomaly detection methods.","marker":"[11]"},{"why":"Explains why AUROC is misleading under class imbalance, which the paper uses to justify preferring Dice.","marker":"[12]"},{"why":"Provides the simpler 2D BrainMRI benchmark against which BraTS-MET is contrasted to explain the performance discrepancy.","marker":"[13]"}],"fun_headline_variants":["CLIP zero-shot fails brain tumor detection in MRI","Zero-shot CLIP not there yet for brain tumors","CLIP-based ZSAD falls short in brain MRI","CLIP zero-shot detection: below clinical bar on MRI","Are CLIP zero-shot models there yet? Not for MRI"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The negative conclusion assumes that the particular evaluation choices—default industrial prompts, unchanged hyperparameters, axial T2 slices, and 2D-to-3D slice stacking—are representative of what CLIP-based zero-shot anomaly detection can achieve on this medical task.","fun_headline_variants_meta":{"raw":{"variants":["CLIP zero-shot fails brain tumor detection in MRI","Zero-shot CLIP not there yet for brain tumors","CLIP-based ZSAD falls short in brain MRI","CLIP zero-shot detection: below clinical bar on MRI","Are CLIP zero-shot models there yet? Not for MRI"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000713,"raw_usage":{"total_tokens":3141,"prompt_tokens":813,"completion_tokens":2328,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":429,"completion_tokens_details":{"reasoning_tokens":2248}},"tokens_in":429,"tokens_out":2328,"duration_ms":16202,"temperature":1.0,"reasoning_tokens":2248,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:46:44.221589+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"If an independent evaluation with specialized metastasis prompts, a 3D-aware aggregation, or a different CLIP-like backbone produced a mean 3D Dice above 0.50 on the same BraTS-MET test split without training on BraTS labels, the claim that CLIP-based ZSAD is not ready would be falsified for that configuration; conversely, reproducing the below-0.50 result under those improved conditions would confirm the paper's conclusion.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides PMC-CLIP, the biomedical-pretrained backbone used as an alternative to the original CLIP."},{"cited_title":"Davis, M","cited_arxiv_id":null,"evidence_quote":"Explains why AUROC is misleading under class imbalance, which the paper uses to justify preferring Dice."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the simpler 2D BrainMRI benchmark against which BraTS-MET is contrasted to explain the performance discrepancy."}],"review_version":1}