{"id":"b8e28056-bd43-4f6c-8ed8-78e4e149773d","arxiv_id":"2607.18325","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Current VLMs tend to label anomalous scenes as hazardous, and a four-way hazard/anomaly benchmark exposes this conflation more clearly than binary safe/unsafe tests.","lead":"This paper asks whether vision-language models confuse 'weird but safe' images with 'actually dangerous' images, and introduces a 610-image dataset that scores hazard and anomaly separately. It reports that current VLMs often treat contextual oddities as physical dangers, and argues that safety evaluations should keep the two concepts apart.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No confusion matrix or conditional error analysis is reported; the headline claim that VLMs conflate anomaly with hazard is asserted but not directly measured.","rationale":"The reader identified the synthetic dataset and non-expert annotations as the weakest assumption, which is a valid external-validity concern. However, my concern is more immediate: even if the dataset were perfectly realistic and well-labeled, the paper's headline claim is not actually demonstrated by the reported metrics. The per-class F1 scores and Hamming loss are aggregate; they cannot reveal the specific error pattern of anomalous images being labeled hazardous. The baseline comparison with the binary dataset maps away the anomaly/hazard distinction and changes the prompt, so it also cannot support the claim. This is an internal-evidence gap. I partially agree with the reader because the baseline confound is mentioned in their rationale, but they did not flag the absence of direct confusion analysis. The dataset and framework may still be useful, and the claimed conflation is plausible, so I would keep the verdict CONDITIONAL rather than REJECT. The authors should provide a confusion-matrix analysis or explicit conditional error rates to substantiate the central claim.","tokens_in":12041,"tokens_out":3088,"duration_ms":36251,"concrete_test":"Compute the full 4x4 confusion matrix for each model on the 610-image dataset (or the balanced subset). Report P(pred=Hazardous | true=Anomalous) and P(pred=Anomalous | true=Hazardous) across all prompting conditions. If these cross-class error rates are not substantially above chance (e.g., not >2x the base rate of that class), the abstract's 'frequently misinterpret' claim is not supported. Additionally, to address the prompt confound, run the BSTS binary prompt from [5] on the same four-class dataset and compare confusion patterns directly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in the abstract is that VLMs frequently misinterpret anomalousness as hazardousness. However, the paper never reports a confusion matrix or any conditional error analysis on the four-class dataset. Per-class precision/recall/F1 (Table I) and Hamming loss (Table II) cannot isolate the anomaly-to-hazard error direction. For example, an anomalous image misclassified as 'both' would lower anomaly recall but not appear as a hazard-specific error. The baseline comparison (Section IV, Baseline Comparison) uses the binary VERI dataset, where anomalous cases are mapped to 'safe' and anomalous-hazardous to 'hazardous'; this mapping collapses the very distinction the paper aims to study. Moreover, the zero-shot prompt used for comparison differs from the BSTS prompt in wording and structure, so the reported p=0.0022 improvement could reflect prompt design rather than the four-class taxonomy. Without a direct measure of how often true anomalous images are labeled hazardous (and vice versa), the 'over-reliance on contextual irregularity' conclusion is an interpretation, not an empirical result.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a four-class hazard/anomaly taxonomy (Safe, Anomalous, Hazardous, Anomalous-Hazardous) to evaluate whether Vision-Language Models (VLMs) can separate contextual irregularity from physical danger. The authors construct a 610-image dataset, evaluate six VLMs under three prompting strategies (zero-shot, few-shot, chain-of-thought) and two input modalities (images and dense captions), and compare against a binary-safety baseline (VERI/BSTS). The central claim is that VLMs frequently misinterpret anomalousness as hazardousness, and that an explicit four-class scheme exposes failure modes hidden by binary evaluations.","tokens_in":12283,"tokens_out":3123,"duration_ms":35850,"significance":"If substantiated, this is a useful contribution: binary safe/unsafe benchmarks cannot distinguish a model that flags a red costume as blood from one that identifies an actual fire hazard, and the proposed taxonomy is a natural corrective. The paper provides a public dataset and a systematic multi-model, multi-prompt evaluation, which are valuable resources for the VLM safety community. However, the headline claim about anomaly-to-hazard conflation is not directly established by the reported metrics, and several methodological choices (fully synthetic images, unspecified generation pipeline, a baseline comparison that confounds prompt design with taxonomy, and the use of Gemini 3 Flash as both caption generator and evaluated model) weaken the empirical foundation. The work is therefore a promising starting point rather than a definitive demonstration.","major_comments":[{"comment":"The central claim that 'VLMs frequently misinterpret anomalousness as hazardousness' is not directly supported by the reported results. Table I reports per-class precision/recall/F1 for anomaly and hazard, and Table II reports Hamming loss; neither isolates the error direction. An anomalous image misclassified as 'Both' or 'Hazardous' would lower anomaly recall or joint accuracy, but the aggregate metrics cannot distinguish that specific conflation from other error patterns. The paper needs a confusion matrix (4x4) or conditional error rates such as P(predict Hazard | true Anomaly) and P(predict Anomaly | true Hazard) to demonstrate the claimed over-reliance on contextual irregularity. Without this, the abstract's main conclusion is an interpretation, not an empirical result.","section":"Abstract / Section IV, Tables I and II"},{"comment":"The dataset is described as 'artificially generated based on textual descriptions,' but the text-to-image model, generation prompts, and any validation that the synthetic images faithfully instantiate the intended hazard/anomaly semantics are not specified. The paper acknowledges dataset size and annotation subjectivity but does not address the synthetic-to-real transfer question. If the generated images contain systematic artifacts (e.g., odd rendering that itself reads as 'anomalous'), the measured conflation may not reflect real deployment. Please document the generation pipeline, show example images per class, and provide some evidence (e.g., human agreement on a random subset) that synthetic images match the intended categories.","section":"Section III, Dataset Design; Section V, Limitations"},{"comment":"The comparison to the BSTS baseline confounds two factors: the four-class taxonomy and the prompt wording. The binary mapping ('anomalous'→'safe', 'anomalous-hazardous'→'hazardous') collapses the very distinction the paper studies, and the zero-shot prompt differs from the BSTS prompt in structure and content. The reported significant improvement (p=0.0022) could therefore reflect prompt phrasing alone. A controlled ablation is needed: e.g., evaluate the same four-class prompt but with only binary labels, or keep the prompt identical except for the removal of the anomaly class, to isolate the effect of the taxonomy.","section":"Section IV, Baseline Comparison; Figure 4"},{"comment":"Gemini 3 Flash generated the dense captions and is also one of the evaluated models. Its caption-conditioned scores may be inflated because the model is effectively reasoning over its own text generation, which could encode the same biases. The paper notes the captions were reviewed but does not address this circularity. Please either generate captions with a model that is not in the evaluation set, or explicitly flag and exclude Gemini 3 Flash's caption-conditioned results when interpreting the comparison.","section":"Section III, Dense Captioning; Section IV, Dense Caption Input"}],"minor_comments":[{"comment":"The annotation process says researchers 'independently labeled' then 'reviewed collectively to reach a unanimous agreement.' No inter-annotator agreement metric is reported. The limitation is acknowledged in Section V, but a Cohen's kappa or similar statistic would strengthen the dataset's credibility.","section":"Section III, Dataset Design"},{"comment":"The symbolic framework (P, Q_T_i, Q_I_i, R_i, c_i) is introduced but not used consistently in the rest of the paper. It can be simplified or tied more explicitly to the prompts in the task-design subsections.","section":"Section III, Formalization"},{"comment":"Table I is dense and lacks a note explaining the subscripts or how 'per-class' treats the 'Both' category. Table II omits a row for Gemini 2.5 Flash with chain-of-thought prompting; the reason ('did not produce valid responses') should be stated in the table caption rather than only in the text.","section":"Tables I and II"},{"comment":"The captions say 'precision vs. recall graph' but do not explain what each point or curve represents (one point per model-prompt combination? a PR curve?). Please clarify the plotting convention.","section":"Figures 2 and 3"},{"comment":"Reference [9] is dated 2026 and some Roboflow dataset URLs are not standard archival references; please ensure URLs are stable or cite DOIs where possible.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The core idea—separating anomaly from hazard in VLM safety evaluation—is timely and the public dataset is a useful resource. However, the paper currently does not provide the direct evidence needed for its headline claim, and the baseline comparison and caption-generation circularity are substantive concerns. The revisions are feasible: add a confusion-matrix/error-direction analysis, document and validate the synthetic dataset generation, control for prompt effects in the baseline comparison, and address the Gemini 3 Flash caption circularity. If the authors cannot supply confusion matrices (e.g., due to data loss), the abstract should be softened to describe the observed performance gap rather than the specific conflation mechanism. I lean major_revision rather than reject because the research direction is sound and the missing pieces are methodological additions, not fundamental flaws."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Three things you should know about arXiv:2607.18325. First, the useful contribution is the four-class Safe/Anomaly/Hazard/Both taxonomy and the 610-image public benchmark; that's a genuine addition to VLM safety evaluation. Second, the headline claim—that VLMs systematically confuse anomalies with hazards—is not actually measured. The paper reports per-class F1 and Hamming loss but never a confusion matrix or a conditional error analysis showing anomalous images being labeled hazardous. Third, the baseline comparison to Choi et al. is confounded: it changes both the taxonomy and the prompt wording, so the claimed improvement could just be prompt design.\n\nWhat the paper does well: the prompting methodology is described in replicable detail, the dataset is public, and the authors are admirably candid about the limitations—small dataset, non-expert annotators without consensus metrics, synthetic images. They also show a consistent pattern where anomaly F1 is lower than hazard F1 across models and prompts, which is suggestive even if not conclusive.\n\nThe soft spots are real. The missing confusion matrix is the biggest one; without it, the abstract's 'over-reliance on contextual irregularity as a proxy for danger' reads as interpretation rather than result. An anomaly misclassified as 'both' or 'safe' would produce the same low anomaly recall but wouldn't support the conflation story. The baseline comparison is also weak: the BSTS baseline uses a different prompt, and the mapping of anomalies to 'safe' collapses the very distinction the paper is about. The synthetic image generation is underspecified—no model name, no prompts, no validation against real images—so the transferability of results is unknown. The dense captions were generated by Gemini 3 Flash, which is also one of the evaluated models; that's a minor circularity, and the authors acknowledge it.\n\nWho gets value: anyone building or evaluating VLMs for safety-critical HRI or emergency response, particularly when designing alerting systems that need to distinguish unusual from dangerous. The benchmark gives a concrete way to test that distinction, and the four-class scheme is worth adopting as a diagnostic.\n\nRecommendation: send it to peer review. The framework is worth engaging with, and the central claim can be fixed with a proper error analysis. The authors should be asked to add a confusion matrix or conditional error table, and to either match the baseline prompt or present the prompt-vs-taxonomy confound as a limitation rather than a headline result.","headline":"Useful four-class benchmark, but the conflation claim needs a confusion matrix before it can be believed.","tokens_in":12723,"tokens_out":3338,"would_cite":true,"duration_ms":31537,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"VLMs routinely mistake unusual scenes for dangerous ones, a study of six vision-language models shows.","keywords":["vision-language models","hazard detection","anomaly detection","safety reasoning","four-class classification","over-reaction problem","prompt engineering","situational awareness"],"falsifier":"A direct test would be to run the same four-class evaluation on a set of real photographs of safety-critical scenes (e.g., from construction sites or emergency response footage) with validated ground truth from domain experts, and check whether the measured anomaly/hazard conflation persists or vanishes. If the conflation disappears on real images, the synthetic dataset was the cause.","tokens_in":11954,"feed_emoji":"⚠️","tokens_out":1147,"duration_ms":15373,"temperature":0.7,"pith_summary":"This paper argues that current evaluations of vision-language models (VLMs) for safety are too coarse: they ask models to judge scenes as simply safe or unsafe, which hides a systematic failure mode. The authors introduce a four-class taxonomy—Safe, Anomalous, Hazardous, and Anomalous-Hazardous—and evaluate six VLMs across several prompting strategies on a new 610-image benchmark. Their central finding is that VLMs frequently classify merely anomalous scenes as hazardous, revealing an over-reliance on contextual irregularity as a proxy for danger. The paper also shows that explicitly separating anomaly from hazard yields more informative evaluation and can improve calibration, with the best joint classification still far from reliable for safety-critical deployment.","feed_headline":"VLMs confuse unusual scenes with dangerous ones","feed_subtitle":"Six models repeatedly label benign anomalies as hazards; a four-class test exposes the bias.","key_machinery":"The central object is a two-bit label scheme (hazard × anomaly) that treats danger and contextual irregularity as orthogonal dimensions, formalized as classification codes 00 (Safe), 10 (Anomalous), 01 (Hazardous), 11 (Anomalous-Hazardous). The key mechanism is the prompt structure that explicitly defines \"hazard\" (OSHA-grounded) and \"anomaly\" (context violation) and instructs the model to decide each label independently. This orthogonal labeling allows the authors to separate object-level risk cues from contextual deviations, and to measure the over-reaction problem directly.","core_discovery":"The paper's core claim is that state-of-the-art VLMs conflate anomalousness with hazardousness under binary safety prompting, treating unusual context as if it were physical danger. By forcing models to make two independent binary judgments—one for hazard (potential immediate physical harm) and one for anomaly (violation of expected spatial/contextual relations)—the authors demonstrate that this conflation is measurable and consistent across models and prompts. They find that per-class anomaly F1 scores are consistently lower than hazard F1 scores, and that joint classification (four classes) has a best Hamming loss of 0.23, meaning models still mislabel nearly a quarter of scenes even in th","pith_inferences":["The observed conflation of anomaly and hazard may be a general property of large pretrained models, not limited to these six; if so, any VLM used in situational-awareness tasks should be audited with a four-class rather than binary evaluation.","The authors' finding that anomaly classification is consistently harder than hazard classification could be exploited by calibrating decision thresholds differently for the two axes, or by training on more nuanced anomaly examples.","The weak agreement among human annotators on borderline cases suggests that even human ground truth for anomaly vs. hazard is not crisp; a probabilistic labeling scheme (e.g., crowd-sourced with consensus measures) might yield a more reliable benchmark and potentially change which models appear best.","The paper's implication that forcing a four-class taxonomy improves even binary outcomes hints that richer task decomposition could be a general prompt-engineering strategy for safety judgments, worth testing on other binary safety tasks."],"forward_implications":["Binary safe/unsafe safety evaluations of VLMs systematically overstate danger in unusual-but-benign scenes, so deployment in high-risk environments could produce false alarms that distract human operators.","Explicitly separating anomaly from hazard during prompting improves calibration and yields better per-class performance than a single safety judgment, even when the final decision is collapsed to binary.","Dense captions as input do not improve classification over images; they often degrade it, suggesting that current VLMs lose crucial visual context when reasoning from text alone.","Zero-shot prompting with the four-class taxonomy significantly outperforms a prior binary prompt on the VERI benchmark (t-test p=0.0022), indicating that the taxonomy itself aids reasoning.","No tested model, prompt, or input modality achieves joint four-class accuracy sufficient for safety-critical deployment; the best Hamming loss is still 0.23."],"fun_headline_variants":["VLMs see anomalies as hazards","VLMs mistake unusual for dangerous","Model bias: anomaly equals hazard","Four-class test reveals VLM safety flaw","VLMs overreact to odd scenes"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The study assumes that artificially generated images, created from text descriptions without a specified generation model or validation against real footage, faithfully represent the visual semantics of real hazard and anomaly scenes.","fun_headline_variants_meta":{"raw":{"variants":["VLMs see anomalies as hazards","VLMs mistake unusual for dangerous","Model bias: anomaly equals hazard","Four-class test reveals VLM safety flaw","VLMs overreact to odd scenes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000189,"raw_usage":{"total_tokens":1167,"prompt_tokens":733,"completion_tokens":434,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":477,"completion_tokens_details":{"reasoning_tokens":375}},"tokens_in":477,"tokens_out":434,"duration_ms":4242,"temperature":1.0,"reasoning_tokens":375,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T20:28:50.822919+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test would be to run the same four-class evaluation on a set of real photographs of safety-critical scenes (e.g., from construction sites or emergency response footage) with validated ground truth from domain experts, and check whether the measured anomaly/hazard conflation persists or vanishes. If the conflation disappears on real images, the synthetic dataset was the cause.","supporting_citations":[],"review_version":1}