{"id":"666f743a-563d-42b7-8213-eddc3a9b0f68","arxiv_id":"2505.16674","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"ChatGPT-4o, prompted with prior knowledge about normal battery temperature patterns, detects battery thermal anomalies at 86.6% AUC without training, though most trained methods score higher.","lead":"Researchers tested whether general-purpose visual question answering models can spot defects in battery thermal images without any battery-specific training data. The best prompt scored 86.6% AUC, below most trained specialists but at zero training cost, and the study shows strong sensitivity to prompt wording and trial randomness.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central 86.6% AUC in Table III is computed by an unspecified procedure from binary a/b answers, leaving the headline 'competitive performance' claim unverifiable.","rationale":"The reader identified the color-grounding assumption and the selective reporting as key weaknesses. Both are real, but the most load-bearing concern is the undefined AUC metric, because 'competitive performance compared to state-of-the-art' is the paper's stated contribution and is quantified only by that AUC. A correct but unspecified scoring rule could coincidentally support the claim; an arbitrary rule cannot be checked. This is not a matter of interpretation but of missing information essential to evaluation. The paper is honest about trial variation and unsure cases, and the qualitative analysis is useful, but without the AUC procedure, the headline result is not reproducible. I therefore recommend the verdict move from CONDITIONAL to UNVERDICTED: the central claim cannot be adjudicated until the authors disclose and justify the AUC computation.","tokens_in":8978,"tokens_out":5008,"duration_ms":40494,"concrete_test":"Request the raw ChatGPT-4o Prompt 2 outputs for all five trials on all 60 test images. Ask the authors to state the exact score assigned to each image (e.g., fraction of 'No' votes, with a specified rule for 'unsure' responses) and the code used to compute AUC. Independently recompute the ROC from these scores. If the recomputed AUC is not 86.6% or if no defined score yields that value, Table III does not support the central claim. A minimal sanity check is also to verify that the AUC is consistent with the binary accuracies in Table II; e.g., a 82.3% accuracy with 45% normal / 55% anomaly cannot produce an 86.6% AUC unless the score distribution has a particular shape.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"The abstract's central claim of competitive performance rests on a single number: the 86.6% AUC for ChatGPT-4o with Prompt 2 in Table III. The paper never specifies how AUC is obtained from VQA outputs. The method prompts the model for a categorical answer ('a) Yes' or 'b) No'); there is no continuous anomaly score, and the test set has 60 images. The accuracy reported in Table II for that cell is 82.3%, but AUC and accuracy are different metrics that cannot be derived from each other without a score. If the score is the fraction of 'No' votes across five trials, then each image's score is a multiple of 0.2, producing at most 6 distinct values; the resulting ROC is coarse and may over- or under-state performance. If the score is instead taken from model-internal token probabilities, the paper does not say so or how 'unsure' outputs are encoded. Moreover, the SOTA AUCs in Table III are continuous feature-distance scores from trained models; comparing them to an ad hoc vote-based AUC is not valid without a common scoring protocol. Because this number is the only quantitative support for the central claim, the claim is currently unverifiable from the manuscript.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper explores zero-shot anomaly detection in battery thermal images by prompting three VQA models (ChatGPT-4o, LLaVa-13b, BLIP-2) with handcrafted and paraphrased text prompts that encode prior knowledge about normal battery thermal behavior: an even and smooth thermal distribution and a temperature below 50°C. The method is evaluated on the test set of the battery thermal image dataset introduced in [8], using accuracy averaged over repeated trials, per-category accuracies, qualitative output analyses, and a comparison of AUC values against trained state-of-the-art methods taken from [8]. The paper concludes that the zero-shot VQA approach remains competitive despite using no battery-specific training data.","tokens_in":9159,"tokens_out":3429,"duration_ms":31944,"significance":"If the central claim were fully substantiated, the paper would offer a useful and practical direction: avoiding expensive collection of battery thermal training data by leveraging generic VQA models with prompt-based prior knowledge. The study has real strengths: it examines five prompt variants, performs repeated trials, analyzes qualitative reasoning errors (e.g., Figures 4–6), and reports per-anomaly-type accuracy. However, the quantitative evidence for the headline claim is currently incomplete. The 86.6% AUC in Table III is reported without specifying how a continuous score is obtained from categorical yes/no answers, it is selected from one of fifteen model–prompt combinations, and it is not accompanied by confidence intervals or significance tests on a 60-image test set. The very low normal-image accuracy for several prompts also indicates a strong false-positive bias that is not addressed by the AUC comparison. The paper is therefore more of an exploratory study with promising observations than a validated demonstration of competitiveness; the empirical observations are valuable, but the central claim needs substantial additional support.","major_comments":[{"comment":"The computation of the 86.6% AUC is not specified. The method produces categorical answers ('a) Yes' or 'b) No'), and the paper does not explain how these binary responses are converted into a score suitable for an ROC curve. The reported 82.3% accuracy for ChatGPT-4o Prompt 2 cannot be transformed into an AUC without an underlying score. If the score is the fraction of anomaly votes across five trials, the test set admits at most six distinct score values and the resulting ROC is extremely coarse; if the score is obtained from token probabilities or logits, that protocol is never described. Because this AUC is the only quantitative support for the 'competitive performance' claim, the claim is currently unverifiable from the manuscript. Please specify the exact scoring rule, the handling of 'unsure' responses in the score, and whether the ROC is computed per trial or from aggregated scores.","section":"Section IV-C8, Table III"},{"comment":"The comparison with state-of-the-art methods is weakened by selection and lack of uncertainty quantification. ChatGPT-4o with Prompt 2 is chosen from fifteen model–prompt combinations (three models times five prompts) after observing the results, and no confidence interval or significance test is reported for the 86.6% AUC. On a test set of 60 images, differences of a few AUC points are within the range of sampling variability. Reporting the best of fifteen configurations without a pre-specified selection protocol or a multiple-comparison correction makes the headline number difficult to interpret. The authors should either report all fifteen AUC values, provide bootstrap confidence intervals, or describe the selection rule and justify that it does not inflate the reported performance.","section":"Section IV-C8, Table III"},{"comment":"The very low accuracy on normal images undermines the balanced reading of the results. For example, LLaVa-13b with Prompt 2 achieves 2.5% normal-image accuracy, and BLIP-2 with Prompts 1–4 achieves 0% normal-image accuracy, while anomaly accuracy is near 100%. Since the test set has 27 normal and 33 anomalous images, an anomaly-biased classifier can obtain a high overall accuracy and a high AUC despite failing on normal data. The paper should report balanced accuracy, precision-recall curves, or confusion matrices, and should explicitly discuss the safety implications of a detector that flags almost every normal image as anomalous. This issue is load-bearing because the conclusion of 'competitive performance' depends on treating the overall accuracy and AUC as meaningful summaries of detection quality.","section":"Table II, Section IV-C4"},{"comment":"The decision to classify 'unsure' outputs as anomaly predictions is an evaluation choice that favors anomaly detection and can substantially affect the reported numbers. The paper states that this is done for safety, but it does not analyze how sensitive the results are to this choice, nor does it report how many of the final correct predictions in each cell rely on this mapping. Because some LLaVa-13b prompts have up to 41.7% unsure outputs, the treatment of these cases is not a negligible detail. Please report accuracy and AUC both with and without the 'unsure-as-anomaly' rule, or at least provide per-prompt counts of how often the rule changes the final prediction.","section":"Section IV-B and Section IV-C7"}],"minor_comments":[{"comment":"The column header 'After pre-proprecessing' contains a typo; it should be 'After pre-processing'.","section":"Table IV"},{"comment":"The phrase 'one of the way to enhance performance' should read 'one of the ways to enhance performance'.","section":"Section IV-C9"},{"comment":"The notation 'Avg. Acc. (all)' is not explicitly defined. Clarify that it is the accuracy computed over all 60 test images and then averaged across trials, and specify how per-category accuracies are averaged when the category sizes differ.","section":"Tables II and IV"},{"comment":"The abstract claims 'competitive performance compared to state-of-the-art models,' but Table III shows that the zero-shot AUC of 86.6% is below most clean-train SOTA values. Consider tempering the wording to 'comparable to some trained baselines' unless a significance test supports the stronger claim.","section":"Abstract and Section IV-C8"},{"comment":"The color-to-temperature mapping used in the prompts is described only in the figure and Table I. Since the correctness of the entire method depends on the model grounding textual colors to actual pixel colors, it would help to state explicitly that the colormap is the one shown in Figure 1(b) and to discuss any known limitations of VQA models in interpreting colormaps.","section":"Section III, Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for an applied vision or anomaly-detection venue, and the exploratory analysis of prompt sensitivity and trial variability is a useful contribution. The main blocker is the unspecified AUC computation and the lack of uncertainty quantification around the headline 86.6% figure; this is fixable but requires a substantial revision of the evaluation protocol. I would also encourage the authors to reframe the conclusion as an initial exploration rather than a demonstrated competitive result, given the small test set and the strong false-positive bias. The reliance on [8] for both the dataset and the SOTA baselines is acceptable, but the authors should confirm that the comparison is on the identical test split and scoring protocol."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is a small but real exploratory study. The core idea is not new (GPT-4V-AD did zero-shot VQA anomaly detection), but battery thermal images are a new domain, and the paper does useful work: it tests three models and five prompt variants, repeats trials, and openly analyzes the normal-class weakness. The qualitative examples and error analysis are decent. I believe the experiments were run and the Table II numbers are honestly reported.\n\nThe soft spots are real. The central claim—competitive performance versus trained SOTA—rests on a single AUC, 86.6% for ChatGPT-4o with Prompt 2. The paper never says how AUC is computed from the VQA outputs. The method yields categorical answers ('a) Yes' or 'b) No'), and the setup says repeats are averaged. If AUC comes from the fraction of 'No' votes across five trials, each image only has six possible scores, making the ROC coarse and the comparison to continuous feature-distance scores from trained models in Table III apples-to-oranges. If it comes from token probabilities, that should be stated. Either way, the manuscript currently leaves the headline unverifiable. That is a load-bearing gap, since this number is the only quantitative support for 'competitive.'\n\nAlso, the 86.6 is the best of fifteen model-prompt pairs, with no confidence intervals or significance tests. Table III compares against numbers taken from [8], so the protocols are not matched. The normal-class accuracies are low (e.g., 42.2% for ChatGPT-4o Prompt 1), and counting 'unsure' as anomaly is safety-defensible but inflates anomaly accuracy; a sensitivity check would help. The authors admit the normal-data weakness in the conclusion, so they are not hiding it. They ship no code or data, but the test set is public from [8]. I found no circularity; this is a straightforward empirical measurement.\n\nWho is this for? Anyone considering zero-shot VQA for industrial or battery inspection. It is not a validated method, but it is a useful data point on prompt sensitivity and model choice.\n\nRecommendation: send it to peer review. It is a legitimate empirical study, and the missing AUC protocol is fixable with a clear description. If that is clarified and the SOTA comparison made apples-to-apples, it could be a solid workshop-level or short-paper contribution. The abstract currently overstates the result.","headline":"A genuine exploratory study of zero-shot VQA for battery thermal anomalies, with honest reporting but a headline AUC whose computation is left unspecified.","tokens_in":9707,"tokens_out":2207,"would_cite":false,"duration_ms":19325,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A general visual question answering model, given a prompt encoding what a normal battery looks like, detects thermal anomalies at 86.6% AUC without any battery-specific training.","keywords":["zero-shot anomaly detection","visual question answering","thermal imaging","battery safety","prompt engineering","prior knowledge","battery thermal images"],"falsifier":"Take the same test set, feed each image to ChatGPT-4o with Prompt 2, but first ask it to state the highest temperature it sees and which colormap color appears hottest; if its stated temperatures do not track the known ground truth (for example, normal images whose true maximum is below 50°C are described as exceeding 50°C), the color-to-temperature grounding that the method relies on has failed.","tokens_in":8744,"feed_emoji":"🔥","tokens_out":7663,"duration_ms":59258,"temperature":0.7,"pith_summary":"The paper asks whether a general visual question answering model—one that answers text questions about images—can detect battery thermal anomalies without ever seeing battery training data. It encodes prior knowledge about normal batteries, namely a smooth, even heat distribution with temperature below 50°C, into a text prompt that also describes the thermal colormap, and queries ChatGPT-4o, LLaVa-13b, and BLIP-2 on a real battery thermal test set. The best result, ChatGPT-4o with one chosen prompt, reaches 86.6% AUC averaged over five trials, which is competitive with trained state-of-the-art models even though the model was not fine-tuned on battery data. The paper also documents sensitivity to prompt wording, trial-to-trial fluctuation, and a bias toward labeling images as anomalous, with rotation-and-crop preprocessing helping recover accuracy on normal images.","feed_headline":"Untrained vision-language prompt catches battery faults at 86.6% AUC","feed_subtitle":"No training images required—useful where anomaly data is rare or dangerous to collect.","key_machinery":"The central mechanism is a single text prompt engineered to include an introduction naming the image as a battery thermal image, the colormap as an ordered list of color names with its 25–60°C range, and two prior-knowledge criteria for normality—smooth, even thermal distribution without hot or cold spots and temperature under 50°C—followed by a yes/no query about whether the battery is normal. The VQA model must ground those color names in actual pixels and judge both criteria. The same prompt scaffold is rephrased into four variants (Prompts 2–5) to test how sensitive the zero-shot pipeline is to wording.","core_discovery":"The paper's central claim is that zero-shot anomaly detection in battery thermal images is feasible by treating the task as visual question answering, provided the prompt supplies prior knowledge of what normal looks like. The prompt tells the model the colormap (black, blue, cyan, yellow, orange, red, white over 25–60°C) and two normality criteria from the thermal-monitoring literature: an even and smooth distribution with no hot or cold spots, and temperature below 50°C. On the test set of [8], ChatGPT-4o with Prompt 2 reaches 86.6% AUC without any battery-specific training, while LLaVa-13b and BLIP-2 do worse and vary more across prompts; all models detect anomalies more easily than normal images. The authors read this as evidence that VQA's pretrained grounding plus textual prior knowledge can substitute for labeled battery data, with the caveat that prompt selection and repeated trials heavily influence outcomes.","pith_inferences":["The failure pattern suggests the binding constraint is color-to-temperature grounding, not reasoning: if the model cannot map 'red' and 'white' to temperatures above 50°C, no amount of prompt rewording will fix it, and adding a visual colorbar or a few reference examples may help more than more text.","A prompt-ensemble strategy—voting across several rephrasings and trials—could smooth the 11–33 point swings the paper reports and produce a more deployment-ready zero-shot detector.","The same prompt scaffold should transfer to other thermal-monitoring domains with known colormaps, such as fuel cells, electronics, or motors, and this could be tested by reusing the exact prompts on another thermal dataset.","The near-universal success on anomaly classes combined with weak normal accuracy implies the method is usable as a conservative alarm—anything judged abnormal is likely abnormal—but not yet as a clearance tool; a two-stage design with an anomaly-specific reasoning check could address that."],"forward_implications":["A zero-shot VQA pipeline can screen battery thermal images for anomalies without any battery-specific training, removing an expensive and safety-sensitive data collection step.","The best configuration (ChatGPT-4o, Prompt 2) reaches 86.6% AUC, which is below most trained anomaly detectors but competitive given that no battery data is used; the largest gap is on normal images, which are frequently misclassified as anomalous.","Prompt wording acts as a real hyperparameter: averaged accuracy across the whole test set shifts by up to 33 percentage points depending on the prompt and model.","Repeated trials and preprocessing that removes background (rotation plus cropping) improve stability and normal-image accuracy, raising overall performance in several configurations.","Because the method uses no training data, its performance is unaffected by noise in training data, unlike the trained baselines compared in the paper."],"supporting_citations":[{"why":"Supplies the battery thermal image test set, the two prior-knowledge normality criteria, and the trained state-of-the-art baselines that the zero-shot result is compared against.","marker":"[8]"},{"why":"Provides the precedent of using VQA for zero-shot anomaly detection and the practice of adding context to prompts.","marker":"[12]"},{"why":"Is the ChatGPT-4o model whose Prompt 2 output yields the 86.6% AUC central result.","marker":"[14]"},{"why":"Is the LLaVa-13b model used as one of the three VQA comparisons.","marker":"[15]"},{"why":"Is the BLIP-2 model used as a VQA comparison, including its unusual Prompt 5 accuracy.","marker":"[16]"},{"why":"Motivates the prompt-rephrasing analysis by showing that paraphrasing changes prompt-engineering results.","marker":"[18]"}],"fun_headline_variants":["Zero-shot VQA detects battery faults at 86.6% AUC","No training needed: VQA spots battery anomalies","Battery fault detection goes zero-shot with VQA prompts","Ask and detect: VQA finds battery faults without training"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes a general vision-language model can correctly translate the written colormap (color names to temperatures, including the 50°C cutoff) into what it sees in the pixels, and that the two textual criteria—smooth distribution and temperature below 50—fully capture what makes a battery thermal image normal.","fun_headline_variants_meta":{"raw":{"variants":["Zero-shot VQA detects battery faults at 86.6% AUC","No training needed: VQA spots battery anomalies","Battery fault detection goes zero-shot with VQA prompts","Ask and detect: VQA finds battery faults without training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000263,"raw_usage":{"total_tokens":1593,"prompt_tokens":932,"completion_tokens":661,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":548,"completion_tokens_details":{"reasoning_tokens":593}},"tokens_in":548,"tokens_out":661,"duration_ms":5595,"temperature":1.0,"reasoning_tokens":593,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:56:17.748680+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same test set, feed each image to ChatGPT-4o with Prompt 2, but first ask it to state the highest temperature it sees and which colormap color appears hottest; if its stated temperatures do not track the known ground truth (for example, normal images whose true maximum is below 50°C are described as exceeding 50°C), the color-to-temperature grounding that the method relies on has failed.","supporting_citations":[{"cited_title":"Ai-enabled thermal monitoring of commercial (phev) li-ion pouch cells with feature-adapted unsupervised anomaly detection,","cited_arxiv_id":null,"evidence_quote":"Supplies the battery thermal image test set, the two prior-knowledge normality criteria, and the trained state-of-the-art baselines that the zero-shot result is compared against."},{"cited_title":"Gpt-4v-ad: Exploring grounding potential of vqa-oriented gpt- 4v for zero-shot anomaly detection,","cited_arxiv_id":null,"evidence_quote":"Provides the precedent of using VQA for zero-shot anomaly detection and the practice of adding context to prompts."},{"cited_title":"Paraphrase types elicit prompt engineering capabilities,","cited_arxiv_id":null,"evidence_quote":"Motivates the prompt-rephrasing analysis by showing that paraphrasing changes prompt-engineering results."}],"review_version":1}