{"id":"35ea67b6-4124-4af1-948a-c7d29338591a","arxiv_id":"2505.19750","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A training-free anomaly detection method using DINOv2 features and memory-bank nearest-neighbor matching achieves strong F1 scores on MVTec AD 2.","lead":"SuperAD detects defects in industrial images without training, by matching DINOv2 features of a test image against 16 normal reference images. It reports large gains in segmentation F1 over trained baselines on the new MVTec AD 2 benchmark, though the evaluation protocol raises questions.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing ablations and explicit TESTpublic threshold tuning mean the claim that SuperAD 'consistently outperforms' trained baselines is not yet independently verifiable.","rationale":"The reader's weakest assumption is that the hand-set hyperparameters and category-specific post-processing generalize to the private test sets and future categories. My analysis agrees, and I identify the most load-bearing version of that concern: the paper explicitly tunes the decision threshold on TESTpublic and does not ablate the category-specific post-processing or the PCA mask. The reported private-test numbers are consequently not independently verifiable. I do not see a stronger concern: the method itself is coherent, the coreset plus multi-layer DINOv2 features are standard, the PCA-based mask falls out of the described equations, and there is no evidence of data leakage or internal contradiction. The central claim should therefore remain conditional on a more transparent evaluation and, ideally, code release.","tokens_in":8225,"tokens_out":1086,"duration_ms":10551,"concrete_test":"Run an ablation over the category-specific post-processing and the PCA foreground masking on the TESTpublic set: evaluate the 8 categories with and without PCA foreground masking, and with and without closed-region filling. Also sweep the decision threshold on TESTpublic and report the F1 at the max; if the reported numbers require category-specific masking or threshold selection tuned to TESTpublic, the claim of consistency on the private test sets is weakened.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that SuperAD consistently outperforms trained baselines on MVTec AD 2 depends on the reported TESTpriv and TESTpriv,mix segmentation F1 scores. The paper explicitly states that the decision threshold was optimized on TESTpublic. Because TESTpublic was available to the authors, the reported F1 numbers could reflect threshold choice overfit to that set. More importantly, no ablations are reported for the components that likely drive the gains: the greedy coreset selection, the four-layer feature fusion, the PCA foreground masking (applied only to Vial and Wallplugs), and the closed-region filling (applied only to Fabric and Walnuts). The method also depends on a small, hand-set memory bank size of 16 and hand-set resize resolutions. Without ablations or code, the reported 47.8%/43.2% F1 against 21.8%/9.0% for MSFlow is a plausible but unverified result. This does not make the result internally inconsistent, but it does make the central claim load-bearing on undisclosed choices.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SuperAD, a fully training-free anomaly classification and segmentation method for the MVTec AD 2 challenge. The method uses a frozen DINOv2-ViT-L-14 backbone to extract features from four layers (6, 12, 18, 24) of a small memory bank of 16 normal reference images selected by greedy coreset sampling, computes per-layer nearest-neighbor anomaly maps, and averages them into a final segmentation map. Category-specific preprocessing is applied to Vial and Wallplugs (PCA-based foreground masking) and to Fabric and Walnuts (filling closed regions). The authors report segmentation F1 scores on TESTpublic, TESTpriv, and TESTpriv,mix and claim that the method consistently outperforms trained baselines such as EfficientAD, PatchCore, and MSFlow.","tokens_in":8433,"tokens_out":4431,"duration_ms":49938,"significance":"If the reported results are verified, the significance is considerable: a training-free memory-bank method built on DINOv2 features would outperform trained anomaly detectors on a challenging real-world benchmark, with practical advantages in deployment cost and generalization. The paper is clearly written, the method is simple and largely reproducible from the text, and the authors are explicit about the threshold-optimization procedure and about typical failure cases, which is a commendable disclosure. However, the central claim is currently supported only by a single evaluation with undisclosed threshold choices, no ablations, and baseline numbers taken from an external paper; the evidence is therefore not yet sufficient to establish the claimed level of superiority.","major_comments":[{"comment":"The F1 decision threshold is explicitly optimized on TESTpublic. Since F1 is a threshold-dependent metric, the reported TESTpriv values (47.8% and 43.2%) may reflect information leakage from the public test set, and the large gap over MSFlow (21.8% and 9.0%) could be partly an artifact of threshold overfitting. Please report how the threshold was chosen (global or per-category), show the sensitivity of F1 to the threshold, or fix the threshold using a validation split from the training data; also provide error bars if any repeated runs were performed.","section":"§2.2.2, Tables 1–2"},{"comment":"The claim in Section 3.2 that SuperAD 'consistently outperforms previous methods' is contradicted by the paper's own Table 2: on Fruit Jelly for TESTpriv, MSFlow achieves 47.6% while SuperAD achieves 41.3%. This is a load-bearing discrepancy in the central claim. Please revise the claim to reflect average performance or explain the per-category comparison in detail.","section":"Table 2, §3.2"},{"comment":"The reported numbers are internally inconsistent: Section 3.1 states that the official challenge server gives F1 scores of 47.18% and 42.51% on TESTpriv and TESTpriv,mix, while Table 2 lists 47.8% and 43.2% for the same sets. Please clarify which numbers are the official ones, which are locally recomputed, and why they differ; this ambiguity affects the validity of the headline comparison.","section":"§3.1 vs §3.2"},{"comment":"No ablation study isolates the contribution of the core components: the four-layer feature fusion, the greedy coreset selection, the memory bank size of 16, the PCA foreground masking applied only to Vial and Wallplugs, and the closed-region filling applied only to Fabric and Walnuts. The category-specific post-processing appears to have been selected based on observations on the public test set, and without an ablation or a held-out category analysis the reported gains cannot be attributed to the method rather than to the hand-picked post-processing rules.","section":"§2.1.2, §2.2.1, Table 2"}],"minor_comments":[{"comment":"The workshop name appears as 'V AND 3.0' with a spurious space; it should read 'VAND 3.0'.","section":"Title/Author block"},{"comment":"In Equation (3), the left-hand side and the first branch of the right-hand side both use the symbol M_init, which is confusing; please use a different symbol for the corrected mask.","section":"§2.2.1, Eq. (3)"},{"comment":"The name 'Patchcore' should be written as 'PatchCore' to match the reference list and standard usage.","section":"§2.1.2"},{"comment":"The metric 'AU-ROC 0.05' is used without a definition, and the notation later switches to 'AucPro0.05' and 'AU-PRO 0.05'; please define these metrics once and use a single consistent notation.","section":"§2.2.2, §1.2"},{"comment":"The four 'NN Search' blocks in Figure 1 are not labeled with the layer indices (6, 12, 18, 24); adding these labels would make the architecture figure much easier to read.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"This manuscript is a challenge report rather than a full archival paper, so the appropriate bar is whether the reported result is credible enough for the workshop context. The main risk is threshold overfitting to TESTpublic combined with the per-category contradiction on Fruit Jelly; both issues are fixable with additional analysis and a revised claim. I recommend major revision rather than rejection because the method is simple, internally consistent, and potentially correct, but the current evidence is insufficient to support the headline claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this paper claims a training-free DINOv2 memory-bank method beats every trained baseline on MVTec AD 2 by a huge margin in segmentation F1. The method is plausible and the writing is clear, but the evaluation has enough suspect choices that I would not take the headline numbers at face value yet.\n\nWhat is actually new is modest: a few-shot greedy coreset memory bank built from DINOv2-L CLS tokens, features from layers 6/12/18/24, a PCA-based foreground mask for Vial and Wallplugs, and closed-region filling for Fabric and Walnuts. The combination is a natural extension of PatchCore and APRIL-GAN. The paper does not overclaim novelty and is honest about the failure cases. As a challenge technical report, it is fine.\n\nThe soft spots are real. In Section 2.2.2 the authors say they optimize the decision threshold on TESTpublic; that is test-set tuning, and it directly affects the reported F1 numbers. No ablations accompany the PCA mask or the category-specific post-processing, so it is hard to tell which components drive the gains, and the post-processing for Fabric/Walnuts looks like it was added after inspecting failure cases on the public test set. The comparison with baselines uses numbers taken from the MVTec AD 2 paper, and the evaluation protocol (binarization and thresholding) may not be identical. The gap is so large — 47.8 versus 21.8 mean F1 — that I want independent verification before believing it. No code is released, which makes that harder.\n\nThat said, the core idea is coherent and the method is simple enough to reproduce. The claims about being training-free are accurate in the sense of no gradient-based training. The paper is honestly scoped as a workshop report.\n\nI would send this to a serious referee if it were submitted as a paper, but with the expectation that the evaluation is tightened: a proper validation split for threshold selection, ablations for the components, and ideally code release. For a challenge report, it is a useful data point. For a paper claiming consistent dominance over trained methods, it is not yet convincing.\n\nWorth bringing to a reading group if you track anomaly detection benchmarks; I would cite it cautiously as a related result.","headline":"Training-free DINOv2 memory-bank method on MVTec AD 2 looks promising, but the huge F1 numbers are undermined by test-set threshold tuning and category-specific post-processing.","tokens_in":8976,"tokens_out":3010,"would_cite":true,"duration_ms":28370,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A training-free method claims to beat all trained anomaly detectors on MVTec AD 2.","keywords":["anomaly detection","anomaly segmentation","MVTec AD 2","DINOv2","memory bank","nearest neighbor","training-free","industrial inspection"],"falsifier":"Run the identical pipeline on the MVTec AD 2 private protocol with the segmentation threshold fixed across all categories at a single value not derived from TESTpublic labels, and with the Fabric/Walnuts closed-region filling disabled; if the mean F1 then falls below the best trained baseline, the claimed advantage rests on public-set calibration and category-specific heuristics rather than the training-free representation.","tokens_in":8035,"feed_emoji":"🔍","tokens_out":7869,"duration_ms":79140,"temperature":0.7,"pith_summary":"SuperAD claims that a fully training-free procedure can outperform every trained anomaly-detection baseline on the MVTec AD 2 benchmark, which contains transparent, reflective, and occluded industrial objects under varied lighting. The method extracts features from four layers of the pre-trained DINOv2-large vision transformer, compares each test patch against a small memory bank of 16 normal reference images per category chosen by greedy coreset selection, and averages the per-layer nearest-neighbor distance maps into a segmentation. On the private test set it reports a mean segmentation F1 of 47.8% (43.2% on a mixed-lighting set), compared with 21.8% for the best prior method on the standard set. If correct, this would show that the representational power of large self-supervised models alone, without any task-specific training or fine-tuning, is sufficient to handle the physical complexity and lighting shifts that defeat current trained detectors.","feed_headline":"No training needed: SuperAD beats trained anomaly detectors","feed_subtitle":"DINOv2 memory-bank matching scores 47.8% F1 on the private set, versus 21.8% for the best trained model.","key_machinery":"The central mechanism is a memory-bank nearest-neighbor anomaly score computed from multi-scale DINOv2 features. DINOv2 is a self-supervised vision transformer pre-trained on a large unlabeled image corpus; here its ViT-L-14 variant supplies frozen feature maps at four depths. Greedy coreset selection (the PatchCore procedure) picks 16 normal reference images per category so the bank spans the variety of normal appearances. For a test image, each spatial feature vector is matched to its nearest neighbor in the bank at each of the four layers; the distance maps are averaged and upsampled to a final anomaly map. Two additional hand-crafted steps support the core mechanism: a PCA-based binarization of shallow-layer features produces a foreground mask used for Vial and Wallplugs, and closed-region filling is applied for Fabric and Walnuts.","core_discovery":"The central discovery claimed is that nearest-neighbor retrieval against a frozen DINOv2 feature bank is a stronger real-world anomaly segmenter than learned reconstruction, distillation, or flow-based methods when the benchmark contains transparent materials, bulk objects, and lighting variation. For each of the eight categories, 16 normal images are selected greedily to maximize diversity; features from layers 6, 12, 18, and 24 of DINOv2-large are stored as the memory bank. At test time, every patch's nearest-neighbor distance to this bank is computed at each layer, and the four distance maps are averaged and upsampled. The paper attributes the method's robustness under lighting shifts to this matching process: normal regions find similar stored features across lighting conditions, while anomalies do not. Reported per-category gains are large (Can: 17.3% vs 5.0% best prior; Rice: 60.9% vs 19.1%), and the paper additionally reports competitive ClassF1 scores of 70.2% and 74.4% for image-level classification on the two private test sets.","pith_inferences":["The reported margin probably over-attributes the improvement to training-free matching: two categories (Fabric, Walnuts) receive a fill-closed-regions post-processing that assumes anomalies are enclosed by normal-looking structure, and the F1 decision threshold is tuned on the public test set; a fair transfer test would freeze both.","A natural extension the authors do not explore is to test the same pipeline on other anomaly benchmarks (e.g., VisA, BTAD) without category-specific post-processing, which would isolate how much of the win comes from DINOv2 features versus the hand-built heuristics.","The PCA foreground-masking component suggests a cheap way to inject object-level priors into retrieval-based detectors; one could combine it with rough object proposals from a generic segmentation model to avoid per-category manual selection.","The paper itself documents failure cases (bubbles, specular highlights, missing-type anomalies) that bound the claim; a fair reader should view the quoted margins as category-dependent."],"forward_implications":["If these results hold, the primary bottleneck in industrial anomaly detection moves from training algorithms to the quality of pre-trained visual features, since a training-free retriever beats trained detectors.","The small drop from 47.8% to 43.2% under mixed lighting, versus the large drops for trained methods like MSFlow (21.8% to 9.0%), would indicate that nearest-neighbor matching on robust features degrades gracefully under illumination shift.","The per-category memory bank of only 16 normal images implies that few-shot deployment on a new product line may need no gradient training, only a handful of defect-free images and a forward pass.","The near-zero baseline scores on categories such as Wallplugs (0.0%) imply that previously published methods essentially fail at those scenarios, making the reported 13.7% a substantial practical improvement even if far from perfect."],"supporting_citations":[{"why":"Supplies the frozen DINOv2-large backbone whose layer-6/12/18/24 features are the entire input representation for the memory bank and nearest-neighbor matching.","marker":"[9]"},{"why":"Supplies the greedy coreset selection algorithm used to choose the 16 diverse normal reference images per category.","marker":"[10]"},{"why":"Defines the MVTec AD 2 benchmark with its eight categories, lighting splits, and the public/private test protocols; the paper's baselines and comparison numbers come from this dataset's paper.","marker":"[6]"},{"why":"The strongest trained baseline in the comparison table (mean F1 21.8% on TESTpriv, 9.0% on TESTpriv,mix) that SuperAD claims to surpass.","marker":"[14]"},{"why":"A trained baseline (EfficientAD) from the MVTec AD 2 paper that SuperAD outperforms, illustrating that even specialized fast detectors fall behind the training-free method.","marker":"[1]"}],"fun_headline_variants":["SuperAD: No training, pure DINOv2 matching for anomaly detection","SuperAD: memory-bank matching outperforms trained models","Zero-training anomaly detection with DINOv2 memory bank","SuperAD: no training, just DINOv2 nearest-neighbor matching"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The fixed configuration (16 reference images, DINOv2 layers 6/12/18/24, PCA threshold 1.0, 3x3 morphological kernel, and the category-specific foreground-mask and fill-region post-processing, along with an F1 threshold tuned on the public test set) transfers unchanged to the private test sets and to unseen categories; if any of these choices were fitted to the public set, the claimed margin will not generalize.","fun_headline_variants_meta":{"raw":{"variants":["SuperAD: No training, pure DINOv2 matching for anomaly detection","SuperAD: memory-bank matching outperforms trained models","Zero-training anomaly detection with DINOv2 memory bank","SuperAD: no training, just DINOv2 nearest-neighbor matching"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000627,"raw_usage":{"total_tokens":2929,"prompt_tokens":1005,"completion_tokens":1924,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":621,"completion_tokens_details":{"reasoning_tokens":1849}},"tokens_in":621,"tokens_out":1924,"duration_ms":13761,"temperature":1.0,"reasoning_tokens":1849,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:06:36.360890+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the identical pipeline on the MVTec AD 2 private protocol with the segmentation threshold fixed across all categories at a single value not derived from TESTpublic labels, and with the Fabric/Walnuts closed-region filling disabled; if the mean F1 then falls below the best trained baseline, the claimed advantage rests on public-set calibration and category-specific heuristics rather than the training-free representation.","supporting_citations":[{"cited_title":"Towards to- tal recall in industrial anomaly detection","cited_arxiv_id":null,"evidence_quote":"Supplies the greedy coreset selection algorithm used to choose the 16 diverse normal reference images per category."},{"cited_title":"Msflow: Multiscale flow-based framework for unsupervised anomaly detection","cited_arxiv_id":null,"evidence_quote":"The strongest trained baseline in the comparison table (mean F1 21.8% on TESTpriv, 9.0% on TESTpriv,mix) that SuperAD claims to surpass."},{"cited_title":"Efficien- tad: Accurate visual anomaly detection at millisecond-level latencies","cited_arxiv_id":null,"evidence_quote":"A trained baseline (EfficientAD) from the MVTec AD 2 paper that SuperAD outperforms, illustrating that even specialized fast detectors fall behind the training-free method."}],"review_version":1}