{"id":"cb30556c-024a-4c23-8afc-e59e5865a99d","arxiv_id":"2502.09057","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A fine-tuned vision-language model with one in-context example classifies product defects with 0.804 MCC on MVTec AD without per-product retraining, though the evaluation protocol needs scrutiny.","lead":"A vision-language model is fine-tuned on 941 web images of good and defective products, then asked to inspect new products using just one example image and a text description of the criteria. The method reports an MCC of 0.804 on MVTec AD, which would make factory inspection setup far cheaper if the evaluation protocol survives scrutiny.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (1) selects support from 'all images except q', so MVTec test images (the only defective MVTec images) are used as ICL examples; Table 3's '1-pos' confirms test-label leakage. Clean support/query split required.","rationale":"The paper is clearly written and transparent about many limitations, but the central empirical claim rests on the one-shot evaluation of MVTec AD, and that evaluation is not a clean few-shot protocol. The reader's weakest assumption identifies the same load-bearing concern: the support example pool is never specified, and the available evidence strongly indicates it includes test-set images. Eq. (1) literally searches over all images except the query, and because MVTec AD has no defective images in its training split, the '1-pos' ablation in Table 3 requires test-set defective images as ICL demonstrations. That means the model is being given labeled examples from the very distribution it is supposed to generalize to, which is not available when deploying to a new product. The concern is not about disagreement with any external consensus; it is a protocol-level correctness issue that directly invalidates the headline MCC of 0.804 and F1 of 0.950 as evidence for the one-shot claim. The proposed concrete test would resolve the ambiguity: if a clean disjoint support set preserves the performance, the concern is unfounded; if not, the numbers are inflated. I therefore agree with the reader's verdict of reject, and no verdict adjustment is needed.","tokens_in":12972,"tokens_out":5272,"duration_ms":52348,"concrete_test":"Re-run Table 1 and Table 3 under a strictly disjoint protocol. For each MVTec/VisA category, randomly partition the test images into a support pool and a query set; select the support image using Eq. (1) restricted to the support pool (never the query set), and report per-category and aggregate MCC/F1. Also run '1-neg' with support restricted to the MVTec train (non-defective) split, and '1-pos' with defective images from an external held-out source or synthetic anomalies, since MVTec train has no defective images. If MCC/F1 drop materially (e.g., MCC below ~0.5), the headline numbers depend on test-set support leakage. Inspecting the released code for the exact pool in Eq. (1) would also settle this.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The one-shot evaluation protocol is not a true few-shot protocol. In §3.4, Eq. (1) defines the search space as all images except the query (i denotes the index of the image except for q), with no separate support split. §4.1 says MVTec AD and VisA 'were not used at all during training' but never states whether support images are drawn from the training split, a held-out support set, or the test set itself. Since MVTec AD's training split contains only non-defective images, the '1-pos' entries in Table 3 (one defective example) can only come from the test split. Thus the reported one-shot MCC/F1 use test-domain images as labeled demonstrations, and Eq. (1) retrieves a test image nearest to each query. This breaks the central claim that one labeled example plus text is enough for a new product: at deployment, labeled defective examples of a new product are not available, and using test images as support gives the model distributional and label information it would not have. The paper does not report any experiment with a clean support/query split, so the headline MVTec numbers are not evidence for the stated claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a general visual inspection model built on a vision-language model (ViP-LLaVA) that is fine-tuned on a web-collected dataset of non-defective and defective product images. For a new product, the method uses one-shot in-context learning (ICL): an example image (defective or non-defective) is selected by a Euclidean-distance criterion in a ResNet50 feature space, and this image plus an explanatory text prompt is fed to the VLM to classify a query image as defective or non-defective and to output a bounding box. The authors report MCC 0.804 and F1-score 0.950 on MVTec AD in the one-shot setting and also evaluate on VisA, and they provide an ablation comparing their selection algorithm with RICES and with random example selection.","tokens_in":13226,"tokens_out":4393,"duration_ms":45044,"significance":"If validated, the method would offer a practical path to rapid deployment of visual inspection for new products without per-product retraining, and the public release of code and the fine-tuning dataset would be a useful community resource. The use of MCC alongside F1 is a methodological strength, as is the explicit discussion of F1's sensitivity to class imbalance. However, the evaluation protocol as described is compromised by the support-set provenance, and the paper's own reported limitations (poor localization, sensitivity to multi-object scenes, and no benefit over RICES on VisA) substantially temper the claims. The significance of the contribution is therefore contingent on a clean re-evaluation.","major_comments":[{"comment":"The support set for ICL is defined as 'all images except q' with no separate support split, so the one-shot examples are drawn from the same pool as the test queries. In MVTec AD, defective images exist only in the test split, so the '1-pos' entries in Table 3 must use test images as labeled demonstrations. This gives the model access to test-distribution images and their labels, which invalidates the claimed one-shot protocol for new products. The paper must re-run all experiments with a held-out support set that is disjoint from the query set (e.g., a partition of the test set into support and query, or a separate support collection) and report the results under that protocol.","section":"Section 3.4, Eq. (1); Section 4.1; Table 3"},{"comment":"The fine-tuning dataset includes product categories that overlap with the evaluation benchmarks (e.g., bottle, tile, wood, capsules). The paper acknowledges this in Section 4.2.1, but the central claim is about inspecting 'new products' that the model has not been trained for. Evaluation on categories present in the fine-tuning data cannot support the generalization claim. The authors should evaluate on a set of categories that are entirely absent from the fine-tuning data, and report per-category results for those held-out classes.","section":"Section 3.3; Section 4.2.1; Fig. 6"},{"comment":"The paper lists 'detecting defective locations' as a main contribution, but reports a pixel-level AUROC of 0.730 and states that output coordinates 'deviate from the actual defective locations' for all products. This is far below the level expected for visual inspection localization, and the abstract's framing of the method as detecting defective locations is therefore misleading. The localization limitation should be stated explicitly in the abstract and conclusion, not only in the experimental discussion.","section":"Section 4.2.1; contribution 1"},{"comment":"On VisA, the proposed selection algorithm does not improve over RICES: ICL (Ours) gives F1 0.795 and MCC 0.479, while ICL (RICES) gives F1 0.800 and MCC 0.492. The paper states that 'our selection algorithm does not show significant improvement,' but in fact it is slightly worse on both metrics. This weakens the generality of the claim that Eq. (1) is a better selection algorithm, and the paper should either present a corrected comparison under a clean protocol or temper the claim accordingly.","section":"Section 4.2.2; Table 2"}],"minor_comments":[{"comment":"The explanatory texts used as inspection criteria are described only as 'explanatory texts' with no detail on how they are generated or whether they are hand-crafted per product for the evaluation benchmarks. If they are manually written for each MVTec/VisA category, that is important additional supervision that must be disclosed, and an ablation without these texts (or with automatically generated texts) would clarify the contribution.","section":"Section 4.1; Fig. 4"},{"comment":"Many entries are listed as 'N/A' for F1 or MCC. The caption says 'N/A means that zero division occurred,' but it would be useful to state explicitly which condition causes the denominator to be zero (e.g., all predictions in one class) and to confirm that the aggregate row excludes those categories rather than treating N/A as zero.","section":"Table 1; Table 3"},{"comment":"The product list contains typos: 'sadle' should be 'saddle' and 'wodden wheel' should be 'wooden wheel'. Also, the appendix says the training images 'will be publicly available' in future tense, which is inconsistent with the data-release claim in the abstract; please clarify the current availability status.","section":"Appendix, Fig. 8"},{"comment":"Several references contain formatting errors or incomplete author names (e.g., 'Rosias, L.' in the Agarwal et al. entry, and the Gössgens et al. entry has an unusual volume/page field). Please proofread the reference list.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a practical problem and the authors are unusually candid about their limitations, which is commendable. The main concern is the evaluation protocol: the support-set leakage is a load-bearing flaw that can be fixed by a clean support/query split, and the category overlap with the fine-tuning set also needs a disjoint evaluation. If the authors re-run the experiments under a correct protocol and the numbers remain competitive, the paper would be a reasonable contribution. If the corrected numbers are much lower, a rejection would be appropriate. I recommend major revision rather than outright rejection because the experimental design is fixable within the scope of a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague—\n\nThe one thing to know about this paper: the headline 'one-shot' result is not a clean one-shot result. Equation (1) selects the support example from all images except the query, with no separate support set, and MVTec AD's defective images exist only in the test split. That means the '1-pos' entries in Table 3 are test images used as demonstrations. The model sees a labeled defective image from the same distribution as the query, which is exactly the information the paper claims to do without. The stress-test note is right.\n\nThe paper does have genuine value. The combination—fine-tuning a VLM on a small web-collected defect dataset, then using ICL with an explanatory text and a distance-based exemplar selector—is not something I have seen exactly this way. The dataset of 941 images with coordinate annotations could be a useful resource if released. The authors are also refreshingly candid about localization being weak (pixel AUROC 0.730) and about the selection algorithm not beating RICES on VisA.\n\nThe soft spots are not just the leakage. There are no repeated trials or confidence intervals, so the high MVTec numbers are single-run estimates. The fine-tuning categories overlap with benchmark categories (bottle, tile, cable, wood, etc.); the authors argue appearance differs, but that is a judgment call, not a clean separation. The selection algorithm's advantage over RICES is modest and inconsistent across datasets.\n\nI would send this to peer review, because the core idea is worth discussing and the protocol flaw is fixable. But the paper as written should not pass. A referee should demand a clean support/query split, ideally evaluating on held-out categories that are absent from the fine-tuning set, and some error bars. The dataset alone, if released, might justify a shorter dataset paper.\n\nThis is a paper for people working on VLMs for anomaly detection or industrial inspection. It is not a finished claim about a deployable inspection model, but it is a reasonable research direction with an honest write-up.","headline":"Promising combination, but the one-shot evaluation leaks test labels into the support set, so the headline numbers should not be trusted.","tokens_in":13763,"tokens_out":3851,"would_cite":false,"duration_ms":31197,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"One labeled example plus a text criterion lets a vision-language model inspect new products and reach MCC 0.804 on MVTec AD without retraining.","keywords":["visual inspection","vision-language model","in-context learning","few-shot anomaly detection","defect localization","example selection","MVTec AD"],"falsifier":"Run the same one-shot experiment with the support pool restricted to an explicitly disjoint training split, or for defective supports to a separate held-out set, and compare MCC; if the score drops well below 0.804, the reported gain depends on test-set label leakage rather than on the model's inspection ability.","tokens_in":12785,"feed_emoji":"🔍","tokens_out":9812,"duration_ms":88126,"temperature":0.7,"pith_summary":"This paper tries to establish that a single labeled example, one image of a defective or non-defective product plus a one-line inspection criterion, is enough to let a vision-language model examine a brand-new product type. The authors fine-tune ViP-LLaVA on 941 web-scraped images of defective and non-defective products formatted as a unified question-answer task, then switch to in-context learning at inference so that no per-product retraining or threshold tuning is needed. On MVTec AD the one-shot model reaches MCC 0.804 and F1-score 0.950, and on VisA it reaches MCC 0.479 and F1-score 0.795. If true, the method would let inspection systems adapt to new products by editing a prompt rather than collecting a training set.","feed_headline":"One example image plus text can inspect new products without retraining","feed_subtitle":"MCC 0.804 on MVTec AD with one support example and a text criterion, no per-product training.","key_machinery":"The key machinery is the in-context-learning prompt built from ViP-LLaVA's visual-prompt ability. ViP-LLaVA is a vision-language model derived from LLaVA-1.5 that has been trained to follow overlaid red circles or arrows, so it can attend to a marked region in the support image. The paper fine-tunes ViP-LLaVA on a unified VQA dataset whose answers are either \\\"None\\\" or a bounding box, teaching the model the inspection task itself. At inference, the support image and a short text criterion are placed together with the test image in one prompt, and the example is chosen by minimizing the squared Euclidean distance between ResNet50 features of the candidate and the test image, as in $\\| f(x_i) - f(x_q) \\|_2^2$ in Eq. (1), rather than by cosine similarity. This selection rule is what turns a generic fine-tuned inspector into a product-specific one without parameter updates.","core_discovery":"The central claim is that in-context learning with a single support example makes a vision-language model a general visual inspector. The authors take ViP-LLaVA, a VLM that understands visual prompts like red circles and arrows, and fine-tune it on a small self-collected dataset of 941 web images covering 84 product categories, with answers formatted as coordinates of the defective region or \\\"None\\\". At test time, the model is given one example image marked with a visual prompt, an explanatory text saying what counts as a defect, and the test image; the example is selected as the nearest neighbor of the test image in a ResNet50 embedding space. This procedure reaches MCC 0.804 and F1-score 0.950 on MVTec AD in a one-shot manner, and the paper reports it also improves over giving no example and over the RICES cosine-similarity selection. The paper itself notes that defect-localization accuracy is still low, with pixel-level AUROC 0.730, and that multi-object scenes and some categories like Hazelnut remain difficult.","pith_inferences":["A direct next test is to hold the support image fixed and vary only the text criterion; if MCC changes substantially, the text is carrying inspection knowledge and not just class priors.","Since the paper finds that more examples can hurt, a batch-selection rule with a diversity penalty may outperform single-nearest-neighbor selection, and this is testable on the same MVTec AD setup.","In a real production line, the first defective exemplar must come from somewhere; a practical extension is to run the method with only a non-defective support plus text, because the random-support ablation shows 1-neg often beats 1-pos.","Converting the predicted boxes into segmentation proposals, for example by feeding them to a patch-level anomaly scorer, would test whether the coarse localization can be sharpened without retraining."],"forward_implications":["New product lines can be inspected by writing a text criterion and supplying one example image, removing the per-product data-collection and retraining step.","The same checkpoint spans many categories: the method is evaluated on 15 MVTec AD and 12 VisA classes with a single model.","Example choice is decisive: the Euclidean-nearest-neighbor rule beats RICES on MVTec AD (MCC 0.804 versus 0.665), and adding irrelevant examples can reduce accuracy.","The model's answer format includes defect coordinates, so the binary judgment is at least human-auditable even if the localization is too coarse for pixel-level use."],"supporting_citations":[{"why":"Provides ViP-LLaVA, the visual-prompt-capable vision-language model that the paper fine-tunes and uses for inspection.","marker":"(Cai et al., 2024)"},{"why":"Supplies the MVTec AD benchmark used for the headline one-shot evaluation.","marker":"(Bergmann et al., 2019)"},{"why":"Supplies the VisA benchmark used as the second evaluation dataset.","marker":"(Zou et al., 2022)"},{"why":"Defines RICES, the cosine-similarity example-selection baseline the paper compares against and aims to improve.","marker":"(Yang et al., 2022)"},{"why":"Supplies ResNet50, the feature extractor used in the Euclidean-distance example-selection rule of Eq. (1).","marker":"(He et al., 2015)"},{"why":"Supplies the CLIP ViT-L/14 vision encoder that tokenizes images inside ViP-LLaVA.","marker":"(Radford et al., 2021)"},{"why":"Provides the MCC metric the paper argues is more reliable than F1 for imbalanced inspection data.","marker":"(Chicco Davide and Jurman Giuseppe, 2020)"},{"why":"Supplies the F1-score metric used alongside MCC for the reported results.","marker":"(Sokolova et al., 2006)"}],"fun_headline_variants":["One-shot visual inspection, no retraining needed","Vision-language in-context learning for few-shot defect detection","Single example guides new-product inspection","One-shot inspection hits MCC 0.804 on MVTec AD"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the single example image comes from a labeled pool disjoint from the test image, which the paper never states explicitly; on MVTec AD defective images exist only in the test split, so a defective support example would necessarily be test data.","fun_headline_variants_meta":{"raw":{"variants":["One-shot visual inspection, no retraining needed","Vision-language in-context learning for few-shot defect detection","Single example guides new-product inspection","One-shot inspection hits MCC 0.804 on MVTec AD"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000298,"raw_usage":{"total_tokens":1737,"prompt_tokens":968,"completion_tokens":769,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":584,"completion_tokens_details":{"reasoning_tokens":706}},"tokens_in":584,"tokens_out":769,"duration_ms":7759,"temperature":1.0,"reasoning_tokens":706,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T22:46:25.084942+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same one-shot experiment with the support pool restricted to an explicitly disjoint training split, or for defective supports to a separate held-out set, and compare MCC; if the score drops well below 0.804, the reported gain depends on test-set label leakage rather than on the model's inspection ability.","supporting_citations":[{"cited_title":"K., Meyer, G","cited_arxiv_id":null,"evidence_quote":"Provides ViP-LLaVA, the visual-prompt-capable vision-language model that the paper fine-tunes and uses for inspection."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the MVTec AD benchmark used for the headline one-shot evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the VisA benchmark used as the second evaluation dataset."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines RICES, the cosine-similarity example-selection baseline the paper compares against and aims to improve."},{"cited_title":"W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I","cited_arxiv_id":null,"evidence_quote":"Supplies the CLIP ViT-L/14 vision encoder that tokenizes images inside ViP-LLaVA."},{"cited_title":"The advantages of the Matthews correlation coefficient ( MCC ) over F1 score and accuracy in binary classification evaluation","cited_arxiv_id":null,"evidence_quote":"Provides the MCC metric the paper argues is more reliable than F1 for imbalanced inspection data."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the F1-score metric used alongside MCC for the reported results."}],"review_version":1}