{"id":"b3411c23-e106-4fa8-ba6a-9396805d37ae","arxiv_id":"2505.07576","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"On the MIIC semiconductor SEM dataset, feature-based visual anomaly detection methods such as CFA and STFPM achieve strong image-level and pixel-level defect detection, but a reconstruction-based method from 2021 still reports the best image-level scores.","lead":"Researchers compared nine anomaly detection methods on scanning electron microscope images of integrated circuits, using a public semiconductor dataset. The results show that modern feature-based methods find and localize defects well, with CFA and STFPM leading, though an older reconstruction-based method remains competitive at image level.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported feature-based VAD performance may be inflated by checkpoint selection on a validation split that contains anomalous images; the paper must disclose the selection metric to support the unsupervised claim.","rationale":"The reader's verdict emphasizes inherited hyperparameters and reconstruction-baseline fairness. Those are legitimate concerns, but the checkpoint-selection ambiguity is more load-bearing: it potentially invalidates the empirical basis of the main claim. The text in Section 3.4 explicitly describes a validation split drawn from the test set. Given the dataset composition, that split contains anomalies. The common practice of 'saving the best model' uses a performance metric; on an anomaly-containing validation set, the natural metric is AUC/F1, which requires anomaly labels. If that is what was done, the benchmark results are not an unsupervised evaluation. This goes beyond hyperparameter tuning: a supervised model-selection step can systematically favor methods that happen to fit the validation anomalies, and it makes the comparison between methods (as well as the claim of robustness) unreliable until corrected. A concrete re-run with a label-free selection rule would settle the issue. I therefore recommend keeping the conditional verdict, with the added condition that the authors must either disclose label-free selection or correct the protocol.","tokens_in":7396,"tokens_out":9258,"duration_ms":94779,"concrete_test":"Ask the authors to disclose the exact validation metric used to save checkpoints. If it uses anomaly labels, re-run all feature-based methods with a label-free model-selection rule (e.g., last epoch, or lowest training/validation loss on normal images only) and recompute Table 3. If the image-level F1/ROC or pixel-level F1 drops materially (e.g., more than 2 percentage points), the central claim must be qualified; if unchanged, the concern is resolved.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 3.4 states: 'Half of the test set was considered as a validation set for saving the best model during training.' The test set comprises 116 anomalous and 1274 normal images, so the validation subset necessarily contains labeled anomalous images (about 58 anomalies and 637 normals). If the 'best model' is selected on the basis of anomaly-detection performance on this subset, then anomaly labels are used during model selection. That violates the unsupervised premise of VAD and can inflate the reported ROC/F1 numbers for all feature-based methods in Table 3. The paper does not say which validation metric is used, so the reader cannot tell whether the headline result — that pretrained feature extractors are a robust foundation for unsupervised defect detection — reflects the method itself or supervised checkpoint selection. This is the most load-bearing threat to the central claim because it bears directly on the validity of every number that supports the conclusion.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript presents a comparative benchmark of visual anomaly detection (VAD) methods on the MIIC dataset of SEM images of integrated circuits. The authors evaluate three reconstruction-based methods (f-AnoGAN, GANomaly, IAD+Inpainting) and seven feature-based methods (PaDiM, STFPM, CFA, PatchCore, RD4AD, FastFlow, SuperSimpleNet), reporting image-level ROC/F1/PR and pixel-level ROC/F1/PR/PRO. The headline finding is that feature-based methods, especially CFA and STFPM, perform strongly on SEM data despite the domain gap from ImageNet, while a reconstruction-based method (IAD+Inpainting) remains competitive. The paper argues that benchmarks should diversify beyond natural-image datasets and that reconstruction-based methods deserve renewed attention.","tokens_in":7534,"tokens_out":4109,"duration_ms":38276,"significance":"If the reported numbers are valid, this is a useful and timely benchmark. MIIC is the largest public SEM VAD dataset, and the paper is among the first to test modern feature-based methods on it. The inclusion of both image-level and pixel-level metrics, and the explicit comparison with reconstruction-based baselines, addresses a real gap in the literature. The authors also honestly disclose the use of original hyperparameters and the PatchCore subsampling. However, the headline comparisons rest on an unsupervised-protocol question and on the mixing of two evaluation pipelines (Section 3.4 and Table 3), so the significance can only be assessed after those issues are resolved.","major_comments":[{"comment":"The manuscript states that \"Half of the test set was considered as a validation set for saving the best model during training.\" Since the test set contains 116 anomalous images, the validation subset necessarily includes anomalous images. If the best checkpoint is selected using an anomaly-detection metric computed on this subset, then anomalous labels are used during model selection, which contradicts the unsupervised VAD premise and can inflate the feature-based results in Table 3. The paper does not report which validation metric was used. Please disclose the selection criterion; if it is label-dependent, rerun the experiments with a protocol that uses only normal validation data (or last-epoch checkpoints) and report both sets of numbers.","section":"Section 3.4"},{"comment":"The three reconstruction-based results marked with an asterisk are copied from the original MIIC paper, while the feature-based results were produced with the authors' own pipeline (224x224 resizing, ImageNet normalization, PatchCore training on a 50% subset, validation-based checkpointing). A head-to-head comparison of numbers generated under different preprocessing, metric definitions, and model-selection protocols is not a controlled benchmark. The discussion in Section 4.2 explicitly compares IAD+Inpainting (99.3% ROC, 91.2% F1) with CFA, so this conflation is load-bearing for the central claim. Please re-evaluate the reconstruction methods under the same pipeline or substantially weaken the comparative conclusions.","section":"Table 3 and Section 4.2"},{"comment":"The use of all hyperparameters and feature-extraction layers from the original MVTec implementations is disclosed, but no sensitivity analysis is provided. Since the benchmark's ranking and the conclusion that pretrained feature extractors are robust on SEM data are point estimates under these inherited choices, it is possible that some rankings (e.g., PaDiM's low performance) reflect hyperparameter transfer rather than intrinsic method behavior. Please add a sensitivity discussion or justify why the MVTec settings are expected to transfer to MIIC.","section":"Section 3.4"}],"minor_comments":[{"comment":"There is a typo in \"Approaches basedo on generative models\" that should read \"Approaches based on generative models.\"","section":"Section 3.2"},{"comment":"The table lists \"R4AD\" while the text and Table 2 use \"RD4AD\"; please make the acronym consistent.","section":"Table 3"},{"comment":"The verb \"prove\" is too strong for an empirical benchmark; \"demonstrate\" or \"provide evidence\" would be more accurate.","section":"Sections 1 and 5"},{"comment":"The PRO metric is referenced to [20], which appears to be a multi-scale memory autoencoder paper; the PRO metric itself was introduced in earlier work by Bergmann et al. Please cite the original source.","section":"Section 3.3"},{"comment":"The sentence \"the F1 score for anomaly detection tasks is better because it takes into consideration the intrinsic problem of unbalanced datasets\" is vague; F1 is a harmonic mean of precision and recall and is appropriate for imbalanced data, but the current phrasing is imprecise.","section":"Section 4.1"},{"comment":"The claim that MIIC is \"the only large-scale dataset with SEM images\" should be qualified, since NanoTwice also contains SEM images; the distinguishing point is the scale and the integrated-circuit domain.","section":"Section 3.1"},{"comment":"No code or evaluation scripts are made available; for a benchmark paper, releasing the evaluation code would substantially improve reproducibility.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The validation-split concern is the most serious issue and is fixable. I would encourage the editor to require a re-run with a fully unsupervised protocol before considering publication, together with a re-evaluation of the reconstruction baselines under the same pipeline. The paper otherwise fits the journal's scope and the topic is timely."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. This is the first evaluation of seven modern feature-based VAD methods on the MIIC semiconductor SEM dataset, and it does add something real: pixel-level metrics that the original MIIC paper did not report, plus a reasonable discussion of domain shift from ImageNet to SEM. But the headline numbers may be inflated by how validation is done. Section 3.4 says half of the test set is used as a validation set for saving the best model during training. The test set contains 116 anomalous and 1274 normal images, so the validation subset necessarily includes anomalies. If the best model is selected by any anomaly-aware metric on that subset, anomaly labels are entering model selection, and the \"unsupervised\" claim is broken. The paper never states the validation metric. This is not a small footnote: every feature-based number in Table 3 could reflect a supervised early-stopping oracle rather than the method itself. As written, I cannot tell whether CFA's 99% ROC and STFPM's 77% pixel-F1 are genuine unsupervised performance or an artifact of the validation split.\n\nThe paper does good things. It clearly describes the methods, openly marks which results are copied from the original paper, and the direct measurements appear internally consistent. It also makes a fair point about needing benchmarks beyond natural images.\n\nThe mixed pipeline is a real but smaller problem. Three reconstruction-based results are inherited from the original paper, so the head-to-head in Table 3 is not apples-to-apples with the other seven. Also, there is no code, no run-to-run variance, and the MVTec-inherited hyperparameters may not be ideal for SEM. Those lower confidence but do not sink the paper by themselves.\n\nWho is this for? Practitioners in semiconductor inspection and VAD researchers working on domain transfer. It deserves a serious referee, but the referee must force the authors to disclose the validation metric or re-run the experiments with a validation set that excludes anomalies or uses only normal-image statistics. If that gets fixed, the benchmark is a useful reference. As it stands, the central claim is unsupported by clean evidence.\n\nRecommendation: send to peer review with a clear request to address the validation protocol before acceptance. Not a desk reject, but not a pass as-is either.","headline":"Useful first benchmark of feature-based VAD on semiconductor SEM images, but a load-bearing validation-protocol flaw may inflate the headline numbers.","tokens_in":8044,"tokens_out":2523,"would_cite":true,"duration_ms":26959,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Modern unsupervised visual anomaly detection transfers to semiconductor SEM images: pretrained feature extractors reach 99.0% image-level ROC-AUC (CFA) and 77.3% pixel-level F1 (STFPM) on the MIIC benchmark.","keywords":["visual anomaly detection","semiconductor manufacturing","SEM image inspection","MIIC dataset","feature-based anomaly detection","defect localization","unsupervised learning","benchmark"],"falsifier":"Run CFA and STFPM on MIIC and break down pixel-level F1 by defect type across the 116 defective images; if any defect category is consistently missed while aggregate F1 stays high, the claim that pretrained feature extractors are a robust foundation for localization would be false.","tokens_in":7209,"feed_emoji":"🔬","tokens_out":7317,"duration_ms":67455,"temperature":0.7,"pith_summary":"The paper sets out to show that visual anomaly detection, trained only on defect-free SEM images, can find and localize manufacturing defects without any labeled defect samples. It benchmarks seven modern feature-based methods on the MIIC dataset—the largest public SEM dataset for this task—and reports both image-level and pixel-level metrics. If the claim holds, semiconductor inspection systems could be built without costly defect collection, and the common assumption that methods tuned on natural photographs fail on specialized imaging domains would be weakened. The headline evidence is that CFA reaches 99.0% image-level ROC-AUC and STFPM reaches 77.3% pixel-level F1, while all feature-based methods score above 93 on the PRO localization metric.","feed_headline":"Pretrained features spot semiconductor defects at 99% ROC-AUC","feed_subtitle":"On 25k SEM images, CFA leads image-level detection and STFPM leads localization—no defect labels needed.","key_machinery":"The central object is the MIIC benchmark: 25,276 grayscale SEM images of integrated-circuit metal layers, with 25,160 normal images, 116 defective images, and pixel-level ground-truth masks. Against this benchmark the paper runs feature-embedding VAD methods that use neural feature extractors pretrained on natural photographs—teacher–student methods such as STFPM, memory-bank methods such as PatchCore and CFA (coupled-hypersphere feature adaptation), and normalizing-flow methods such as FastFlow—and scores them with ROC, F1, PR, and the per-region-overlap (PRO) metric. The mechanism carrying the argument is domain transfer: features learned from natural images are applied to SEM patches, and the anomaly maps produced by each method are compared against human-annotated defect regions.","core_discovery":"The paper's central claim is that modern feature-based VAD methods work on semiconductor SEM images despite the domain gap from natural photographs. On MIIC, CFA reaches 99.0% image-level ROC-AUC and 87.96% image-level F1, while STFPM reaches 77.30% pixel-level F1, and the authors conclude that pretrained feature extractors remain a robust foundation for defect detection in this field. The paper also reports that the older inpainting-based reconstruction method IAD+Inpainting remains competitive at the image level (99.27% ROC-AUC, 91.23% F1), which the authors read as evidence that reconstruction-based methods deserve renewed attention when large normal-only training sets are available. A notable domain-specific finding is that method rankings differ from natural-image benchmarks: PaDiM underperforms on MIIC, while STFPM outperforms RD4AD, the reverse of their usual ordering.","pith_inferences":["A direct extension of the paper's reasoning is that self-supervised fine-tuning of the feature extractor on unlabeled SEM images should improve feature-based VAD; CFA's feature-adaptation step already points in that direction.","The inherited hyperparameters and feature-extraction layers from natural-image benchmarks mean the reported rankings are likely conservative for feature-based methods; a validation-set search over backbone layers could change which method leads.","Because the reconstruction baselines' numbers are taken from the original publication rather than re-run in the same codebase, a head-to-head re-implementation on the same train-test split would be needed to know whether reconstruction truly matches feature-based methods at scale.","The benchmark covers a single defect class and one metal-layer imaging modality, leaving open whether the conclusions extend to other layers, other process steps, or different defect morphologies in semiconductor fabrication."],"forward_implications":["A semiconductor fab could deploy feature-based VAD without any anomalous training images: the 25,160 normal MIIC images are enough to train detectors that flag the 116 defective images at high ROC-AUC.","Image-level and pixel-level winners differ—CFA leads detection while STFPM leads localization—so deployment choices depend on whether the priority is flagging bad images or telling an operator exactly where the defect is.","The high PRO scores for all feature-based methods imply that anomaly maps align closely with ground-truth defect regions, supporting human-in-the-loop review in an Industry 5.0 setting.","The competitive image-level result of the inpainting baseline suggests reconstruction-based approaches should be re-examined when the normal-only training set is large, rather than being dismissed as obsolete.","Domain-specific evaluation matters: method rankings on SEM data can invert rankings seen on natural-image benchmarks, so benchmark conclusions should not be assumed to transfer across imaging modalities."],"supporting_citations":[{"why":"Supplies the MIIC dataset and the three reconstruction-based baselines whose published results are reused for comparison.","marker":"[5]"},{"why":"Establishes the standard natural-image benchmark from which the methods' hyperparameters and feature layers are inherited.","marker":"[6]"},{"why":"Defines the STFPM teacher–student method that achieves the best pixel-level F1 on MIIC.","marker":"[13]"},{"why":"Defines the CFA coupled-hypersphere feature adaptation method that achieves the best image-level results.","marker":"[17]"},{"why":"Defines the PatchCore memory-bank baseline whose nearest-neighbor patch matching is compared and whose memory requirements motivate subsampling.","marker":"[16]"},{"why":"Defines the FastFlow normalizing-flow baseline used to represent density-based anomaly scoring.","marker":"[19]"},{"why":"Provides the per-region-overlap (PRO) metric used for fair pixel-level localization evaluation across defect sizes.","marker":"[20]"}],"fun_headline_variants":["CFA leads defect detection on unlabeled SEM images","Unsupervised VAD hits 99% ROC-AUC on semiconductor wafers","STFPM beats RD4AD in semiconductor defect localization","Reconstruction methods shine in chip defect detection without labels","Pretrained features tame SEM defect detection with 99% AUC"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark assumes that the hyperparameters and feature-extraction layers taken from the methods' original natural-image implementations are suitable for MIIC without re-tuning.","fun_headline_variants_meta":{"raw":{"variants":["CFA leads defect detection on unlabeled SEM images","Unsupervised VAD hits 99% ROC-AUC on semiconductor wafers","STFPM beats RD4AD in semiconductor defect localization","Reconstruction methods shine in chip defect detection without labels","Pretrained features tame SEM defect detection with 99% AUC"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000497,"raw_usage":{"total_tokens":2379,"prompt_tokens":831,"completion_tokens":1548,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":447,"completion_tokens_details":{"reasoning_tokens":1462}},"tokens_in":447,"tokens_out":1548,"duration_ms":10121,"temperature":1.0,"reasoning_tokens":1462,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:12:27.099542+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run CFA and STFPM on MIIC and break down pixel-level F1 by defect type across the 116 defective images; if any defect category is consistently missed while aggregate F1 stays high, the claim that pretrained feature extractors are a robust foundation for localization would be false.","supporting_citations":[{"cited_title":"Joint anomaly detection and inpainting for microscopy images via deep self-supervised learning","cited_arxiv_id":null,"evidence_quote":"Supplies the MIIC dataset and the three reconstruction-based baselines whose published results are reused for comparison."},{"cited_title":"Mvtec ad — a comprehensive real-world dataset for unsupervised anomaly detection","cited_arxiv_id":null,"evidence_quote":"Establishes the standard natural-image benchmark from which the methods' hyperparameters and feature layers are inherited."},{"cited_title":"Cfa: Coupled-hypersphere-based feature adaptation for target-oriented anomaly localization","cited_arxiv_id":null,"evidence_quote":"Defines the CFA coupled-hypersphere feature adaptation method that achieves the best image-level results."}],"review_version":1}