{"id":"341b328b-49bd-4eb2-af97-8365242e5f8d","arxiv_id":"2508.14504","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"With carefully layered prompts and one or three reference samples, GPT-4.1 detects anomalies in cable images and crimp-force features at F1 levels that PatchCore and Isolation Forest reach only after training on dozens to hundreds of samples.","lead":"PB-IAD is a framework that uses GPT-4.1 with structured natural-language instructions to detect industrial anomalies in images and time-series features without model training or large labeled datasets. The authors report that this prompt-only approach matches or beats classic machine-learning detectors while using only one or a few reference examples, which could make quality control faster to deploy during production ramp-ups.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Test-set-informed prompt refinement undermines the 'solely through semantic instructions' comparison: the decisive Ci/Ei rule was added after observing benchmark failures.","rationale":"The reader's weakest assumption correctly identifies the load-bearing risk: the semantic instructions may be tuned to the test set. The paper's own 'Preliminary tests revealed' sentence is direct evidence of test-guided prompt editing, and the rule it introduces is precisely the kind of decision criterion that changes F1. This is not a disagreement with current consensus; it is an internal protocol gap. The ablations are internally consistent and the datasets are public, which is credit to the authors, but the evidence as presented does not separate generalizable semantic knowledge from benchmark-specific prompt optimization. The appropriate verdict remains CONDITIONAL: the claim is plausible but needs a stricter evaluation protocol before the 'superior performance solely through semantic instructions' framing can be accepted.","tokens_in":21588,"tokens_out":4492,"duration_ms":58028,"concrete_test":"Implement a lockbox protocol: randomly split each benchmark test set into a prompt-development set (e.g., 50%, stratified by defect type) and an evaluation set; use only the development set and the one-shot reference images to author/revise Ci and Ei; freeze the prompts and compute F1 on the evaluation set. Repeat across at least three splits and report mean±std. If fully-instructed one-shot F1 on the held-out evaluation set falls below the current headline by more than ~3 points, or below PatchCore's matched data-efficient performance, the reported gains are partly test-set-informed. A secondary check: run the final locked prompts 10 times (temperature=0) to quantify GPT-4.1 variance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central comparison (one-shot PB-IAD F1=95.7 on MVTec Cable, 91.0 on Stripped Wire, 95.8 on crimp features, versus PatchCore/Isolation Forest requiring 100-190 samples) is only meaningful if the semantic instructions Ci/Ei are fixed domain knowledge, not parameters tuned to the evaluation images. The paper explicitly reports in Scenario 1: 'Preliminary tests revealed that it is crucial to instruct the model that not all anomaly criteria need to be met for a sample to be considered anomalous...' This rule appears in the final Ei prompt ('any single anomaly is grounds for a positive detection') and was added because the model misclassified benchmark images. The same test set was used for prompt development and for the reported F1 scores, while PatchCore's threshold was selected on a separate 20% validation split. The ablation increments (Ci, Ei) therefore measure the effect of prompt editing against the test distribution, not the transferable value of generalizable semantic instructions. The framework may still work, but the headline 'superior performance... achieved solely through semantic instructions' is not supported as an out-of-the-box claim for a new plant from the evidence as presented.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PB-IAD, a prompt-based industrial anomaly detection framework that uses a frozen multimodal foundation model (GPT-4.1) with a structured prompt template (task, context, expertise, output instructions, and optional reference data). The framework is evaluated in three manufacturing scenarios—MVTec Cable visual inspection, a custom stripped-wire dataset with high variability, and crimp-force curve features—with an ablation study that adds context (Ci), expertise (Ei), and reference data (Rd) incrementally. The authors report F1 improvements with each added component, reaching one-shot F1 of 95.7% on MVTec Cable, 91.0% on Stripped Wire, and 95.8% on crimp-force features. They benchmark against PatchCore and Isolation Forest, finding that PB-IAD achieves comparable or higher F1 with one or three reference samples, whereas the classical models need roughly 100–190 training samples. The paper concludes that PB-IAD delivers superior, data-efficient anomaly detection in dynamic, data-sparse manufacturing environments using only semantic instructions and low-shot examples.","tokens_in":21870,"tokens_out":3192,"duration_ms":37567,"significance":"If the reported results hold, the paper makes a valuable empirical contribution: it provides evidence that a frozen multimodal foundation model, guided by structured natural-language instructions, can compete with retrained classical anomaly detectors in low-data industrial settings. The user-centric prompt template addresses a real gap in deploying IAD during ramp-up phases, and the authors have made the custom datasets publicly available, which supports reproducibility and follow-up work. The benchmarking design—plotting classical-model performance as a function of training-set size—is a useful way to quantify data efficiency. However, the central claim of 'superior performance achieved solely through semantic instructions' is weakened by the prompt-tuning procedure and the absence of repeated stochastic evaluations, as detailed below.","major_comments":[{"comment":"The core comparison is compromised by test-set-informed prompt refinement. The paper states: 'Preliminary tests revealed that it is crucial to instruct the model that not all anomaly criteria need to be met for a sample to be considered anomalous...' This rule appears in the final Ei prompt ('any single anomaly is grounds for a positive detection') and was added after observing model behavior on the benchmark images. The same test set was used both for prompt development and for the reported F1 scores, while PatchCore's threshold was selected on a separate 20% validation split. This asymmetry means that the incremental gains from Ci and Ei may partly reflect prompt overfitting to the test distribution rather than generalizable semantic knowledge. The headline claim of out-of-the-box 'superior performance... achieved solely through semantic instructions' therefore needs support from a hel","section":"Scenario 1, ablation study"},{"comment":"All reported metrics come from a single run of a stochastic API model (GPT-4.1). No temperature, sampling parameters, or number of repeated runs are reported. The differences between ablated configurations (e.g., F1 90.1% vs 93.1% in Scenario 1 zero-shot) may be within run-to-run variation, especially given the model's reasoning is sampled. The paper should report mean and standard deviation over multiple runs, and ideally a simple significance test, before claiming that each added instruction component consistently improves performance.","section":"Table 2, ablation study"},{"comment":"The Isolation Forest benchmark selects the 'optimal contamination hyperparameter' from a discrete search space for each training-set size, apparently using the test set to choose the value. This is test-set leakage and gives IF an advantage that is not matched for PB-IAD (whose prompts were also tuned on the test set, but without an equivalent selection protocol). The PatchCore threshold selection on a 20% validation split is appropriate, but the IF protocol should be clarified. If contamination was selected on the test set, the comparison is not apples-to-apples: either use a validation split for IF or report performance across a fixed, pre-specified contamination grid.","section":"Benchmark comparisons, Figs. 9–11"},{"comment":"The framework proposes a prompt pre-processor (GPT-4o) that translates domain-user input into structured prompts, but this module is never ablated or evaluated. The paper's user-centricity claim depends on this module's ability to generate prompts of comparable quality to the manually engineered prompts used in the experiments. Without any evaluation of the pre-processor—e.g., comparing its generated prompts to the hand-written Ci/Ei instructions on the same benchmarks—the framework-level claim is not supported. The authors should either provide such an evaluation or clearly state that the reported results pertain to manually crafted prompts only.","section":"Prompt pre-processing module, Section 3.3"}],"minor_comments":[{"comment":"The table uses German decimal commas (e.g., '97,1 %') while the text uses English decimal points. Please unify the format for clarity.","section":"Table 2"},{"comment":"The conclusion twice refers to 'Isolation Forrest' (typo for 'Isolation Forest'). Also, it states 'utilising GPT-4' while the experiments used GPT-4.1; please correct for consistency.","section":"Conclusion"},{"comment":"The phrase 'achieved solely through semantic instructions' is overstated because the one-shot and few-shot configurations also include reference data. The paper separately acknowledges the contribution of reference samples; the abstract should be reworded to 'semantic instructions and low-shot reference data'.","section":"Abstract and Section 3.2"},{"comment":"The paper states that the model 'is capable of autonomously calculating these features' but the experiments pre-compute the features externally. This capability claim is not demonstrated in the reported results; either provide evidence or qualify the statement.","section":"Scenario 3, experimental setup"},{"comment":"Several figures are referenced with placeholder-like captions (e.g., Fig. 3, Fig. 5) and the images do not appear clearly in the manuscript text. Please ensure figure quality and captions are final.","section":"General presentation"}],"recommendation":"major_revision","confidential_remarks":"The paper is well-positioned for a venue focused on applied industrial AI, and the public release of the custom datasets is a plus. The main risk is that the prompt-tuning protocol conflates development and test data, which, if not addressed, would make the reported margins unverifiable. I would encourage the editor to require a revision that adds a held-out evaluation or clearly reframes the results as an in-the-loop prompt-tuning study, rather than an out-of-the-box comparison."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a useful applied paper with a real weakness in its evaluation design. The PB-IAD framework—layered prompt template (task/context/expertise/output/reference data) around a frozen VLM—is a sensible, user-centric design for low-data industrial anomaly detection, and the paper shows consistent F1 gains as each prompt layer is added across three scenarios: MVTec Cable, their custom stripped-wire set, and crimp-force features. Credit where due: they ship the two custom datasets on Zenodo, they benchmark against PatchCore and Isolation Forest with training-set-size sweeps, and the ablations are internally consistent.\n\nThe soft spot is the one the stress-test flagged, and it lands. The paper reports 'superior performance... achieved solely through semantic instructions,' but the instructions were not fixed in advance. In Scenario 1 they write that 'Preliminary tests revealed that it is crucial to instruct the model that not all anomaly criteria need to be met...' and the final expertise prompt contains the 'any single anomaly is grounds for a positive detection' rule. That rule was added because the model misclassified benchmark images. The same test set that informed the prompt is then used for the headline F1 numbers. That turns the Ci/Ei ablation from a clean test of transferable semantic knowledge into a measure of how well the prompt was fitted to the test distribution. PatchCore had its threshold selected on a separate 20% validation split, so the comparison is asymmetric in a way that flatters the VLM. The framework may still work in a new plant—the one-shot advantage over PatchCore is plausible—but the out-of-the-box claim is not supported as presented.\n\nOther issues are smaller. Single runs of a stochastic API model, no error bars. The proposed prompt pre-processor is never evaluated. And there are no direct baselines against the most related zero-shot VLM methods (WinCLIP, ALFA, the Schiele et al. one-shot GPT-4V result they cite), so we can't tell whether PB-IAD adds anything beyond known prompting tricks.\n\nVerdict: this deserves a serious referee. The fix is straightforward: split the data, develop prompts on one part, evaluate on held-out data, and report variance across a few API runs. As submitted I wouldn't take the 'superior performance' framing at face value, but the core idea is sound and the datasets are a real asset.","headline":"Plausible prompt-only industrial anomaly detection framework, but the central claim is undercut by test-set-informed prompt tuning and single-run metrics; worth review with a held-out prompt split.","tokens_in":22317,"tokens_out":3346,"would_cite":false,"duration_ms":31633,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A frozen multimodal foundation model, prompted only with semantic instructions and one reference image, matches or beats retrained classical detectors in data-sparse manufacturing.","keywords":["industrial anomaly detection","multimodal foundation models","in-context learning","prompt engineering","low-shot learning","quality control","visual inspection","time-series anomaly detection"],"falsifier":"A clean test would start with a fresh product variant and a domain expert who writes the context and expertise instructions from a written quality specification before seeing any test images. Freeze those instructions, run PB-IAD on the new variant, and compare it with PatchCore trained on increasing numbers of normal samples. If the frozen prompt loses most of its advantage, or PatchCore matches it with far fewer than 100 samples, the reported gains depend on prompt refinement against benchmark images rather than on semantic instructions alone.","tokens_in":21500,"feed_emoji":"🔍","tokens_out":9367,"duration_ms":98998,"temperature":0.7,"pith_summary":"PB-IAD proposes that industrial anomaly detection can be initialized from zero or one example by treating a frozen multimodal foundation model as the detector and encoding all domain knowledge as prompt text. The paper tests this on three manufacturing scenarios—cable cross-section images, stripped-wire images with high variability, and crimp-force curve features—using GPT-4.1 and an ablation that adds context, expertise, and reference samples one at a time. In every scenario the fully instructed prompt reaches F1 scores around 91 to 96 percent, and classical detectors such as PatchCore and Isolation Forest need on the order of one hundred or more training samples to match it. The practical point is that a quality-control system could be deployed at ramp-up before defect data accumulates, and updated by plant experts editing natural-language instructions instead of retraining models.","feed_headline":"One frozen model beats retrained detectors on sparse shop-floor data","feed_subtitle":"Instructed one-shot detection hits F1 95.7; classical baselines need 100-190 samples to catch up.","key_machinery":"The central object is the PB-IAD prompt template, a modular instruction stack with five sections: task instruction, context instruction, expertise instruction, output instruction, and optional reference data. A separate prompt pre-processor turns a domain expert's natural-language input into those sections, so the template can be populated without data-science expertise. The detector is a frozen multimodal foundation model (GPT-4.1) doing pure in-context learning: no weights are updated, no training set is required, and the only data-dependent input is one or a few reference samples. The machinery's job is to convert tacit process knowledge into decision boundaries the model can apply; a cri","core_discovery":"On the paper's own terms, the central discovery is that semantic instruction depth substitutes for training data in industrial anomaly detection. Using a five-section prompt (task, context, expertise, output, optional reference images) and frozen GPT-4.1, the one-shot fully instructed configuration reaches F1 95.7% on MVTec Cable, 91.0% on a high-variability stripped-wire set, and 95.8% on crimp-force features; zero-shot with full instructions reaches 93.1% and 87.9% on the first two. The ablation shows each added section—general context, then expert rules—raises recall substantially, while a single reference image adds less when instructions are already detailed. PatchCore needs roughly 100","pith_inferences":["The paper leaves open whether the AND-vs-OR rule it discovered ('not all criteria need to be met') is specific to GPT-4.1 or a general vision-language model bias; testing the same prompts on other models would settle that.","Because one-shot binary examples sometimes made the model compare too literally, the model's own reasoning text could be used to detect and correct over-reliance on reference images.","The practical value depends on prompt transferability; a direct test would freeze context and expertise text written from a written specification and run it on a new product variant without test-image feedback.","If the reasoning output is consistent, it could double as an audit log, letting quality teams cluster false alarms by stated reason instead of treating the detector as a black box."],"forward_implications":["Quality control can begin at ramp-up with zero or one reference sample, while memory-bank detectors like PatchCore need roughly 100-190 normal samples to match.","Domain experts can update the anomaly definition by editing the prompt, so new defect types or product variants do not require retraining.","Detailed semantic instructions matter more than additional reference images: the zero-shot to one-shot gap shrinks as instructions are enriched.","The same template covers both visual and time-series feature modalities, unifying product inspection and process monitoring in one configuration.","Adding an anomalous reference image can hurt recall compared with using only a normal reference, so more class examples are not automatically better."],"supporting_citations":[{"why":"Supplies the MVTec Cable images used in Scenario 1, the stable-condition benchmark.","marker":"[51]"},{"why":"Provides the Stripped Wire dataset used in Scenario 2, whose high variability and subtle defects define the dynamic-condition test.","marker":"[53]"},{"why":"Provides the Crimp Force Curve dataset used in Scenario 3 for the time-series feature modality.","marker":"[54]"},{"why":"Defines the PatchCore baseline whose training-sample requirements are compared with PB-IAD in the two visual scenarios.","marker":"[58]"},{"why":"Defines the Isolation Forest baseline used to benchmark the time-series scenario.","marker":"[59]"},{"why":"Establishes in-context learning as the parameter-free adaptation paradigm the framework relies on.","marker":"[33]"},{"why":"Grounds the foundation-model family (GPT-4) whose multimodal reasoning PB-IAD uses.","marker":"[20]"}],"fun_headline_variants":["Frozen AI beats retrained detectors with just one image sample","Semantic prompts match 190-sample detectors using one sample","One-shot detection with GPT-4.1 tops data-hungry PatchCore","Language-driven anomaly detection needs one image, not 100","Single sample beats classical baselines in industrial defect detection"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The load-bearing premise is that the refined context and expertise instructions are general process knowledge rather than wording tuned against the benchmark images; if that premise fails, the 'solely through semantic instructions' advantage partly reflects prompt optimization.","fun_headline_variants_meta":{"raw":{"variants":["Frozen AI beats retrained detectors with just one image sample","Semantic prompts match 190-sample detectors using one sample","One-shot detection with GPT-4.1 tops data-hungry PatchCore","Language-driven anomaly detection needs one image, not 100","Single sample beats classical baselines in industrial defect detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000835,"raw_usage":{"total_tokens":3508,"prompt_tokens":802,"completion_tokens":2706,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":2620}},"tokens_in":546,"tokens_out":2706,"duration_ms":21735,"temperature":1.0,"reasoning_tokens":2620,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T18:29:05.342122+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A clean test would start with a fresh product variant and a domain expert who writes the context and expertise instructions from a written quality specification before seeing any test images. Freeze those instructions, run PB-IAD on the new variant, and compare it with PatchCore trained on increasing numbers of normal samples. If the frozen prompt loses most of its advantage, or PatchCore matches it with far fewer than 100 samples, the reported gains depend on prompt refinement against benchmark images rather than on semantic instructions alone.","supporting_citations":[{"cited_title":"Scheck, Stripped Wire Dataset, Zenodo, 2025","cited_arxiv_id":null,"evidence_quote":"Provides the Stripped Wire dataset used in Scenario 2, whose high variability and subtle defects define the dynamic-condition test."}],"review_version":1}