{"id":"796fe4f3-c576-4448-8423-7e3e25b6be4d","arxiv_id":"2603.03989","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Vision-language models confidently misread face-like objects as people, while pure-vision classifiers stay diffuse and detectors suppress responses, showing uncertainty and bias are independent.","lead":"Six computer-vision models were shown pictures of objects that look like faces, to test how they handle ambiguous evidence. The result: different model families react in opposite ways—some confidently call the objects “human,” others stay vague or stay silent—so a confident answer is not a safe answer.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central decoupling claim rests on RAI averaged over unmatched coverage: detectors' 'low uncertainty' may reflect selective firing on easy regions, not representation; the paper never tests thresholds to support 'representation over thresholds.'","rationale":"The reader's CONDITIONAL verdict is appropriate, but the most load-bearing weakness is not the prompt/alignment disclosure per se—it is the selection-bias in how RAI is computed across models with drastically different detection rates. The paper's strongest claim is the decoupling of uncertainty and bias, and Figure 8 is its key evidence. However, RAI (Eq. 4) is computed only over regions that were matched to a prediction. Box-level classifiers are forced to classify every annotated crop, so their RAI includes hard, ambiguous cases. Detectors only fire on a small, likely easy subset (YOLOv8 ~40%, RetinaFace ~0%), so their low RAI may simply reflect that they respond only when confident. This is not a representational property; it is a consequence of coverage. The GT-box control in §4.5 tries to factor out localization but only reports response rate and mean Human score, not a five-way distribution on all boxes, so it cannot rescue the RAI comparison. If the test I propose shows detector RAI is high on the full GT-box set, the paper's central decoupling claim fails. The threshold claim is also untested: no experiment varies thresholds, so 'more representation than thresholds' is an interpretive overlay rather than a result. The reader's concern about undisclosed prompts/mappings is real but secondary; it affects the magnitude of LLaVA's bias, whereas the RAI coverage issue threatens the core conceptual contribution. I therefore recommend keeping the conditional acceptance, but the required revision must include a fairness-controlled RAI analysis, not merely prompt disclosure.","tokens_in":9470,"tokens_out":7946,"duration_ms":75126,"concrete_test":"Recompute RAI for all six models on the identically-cropped GT boxes used in §4.5. For detectors, obtain a five-way distribution on every box even when no detection is produced: e.g., run YOLOv8/RetinaFace at a very low confidence threshold and, if still no detection, impute a uniform distribution (or a background class). Then redraw Fig. 8. If detector RAI rises to ViT-like levels on ambiguous boxes, the reported uncertainty–bias decoupling collapses. Also release LLaVA's exact prompt and parsing; if alternative prompts change its entropy, the low-RAI claim is prompt-dependent.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Equation (4) defines RAI as the average entropy over matched regions. For box-level models (CLIP, LLaVA, ViT) matched regions are all annotated crops, so RAI averages over easy and hard cases alike. For detectors, a region enters RAI only if the model produces a matched detection. Since YOLOv8 matches only ~40% of primary regions and RetinaFace ~0% (Fig. 4), their RAI is computed on a small, easy subset—likely the most face-like instances. The claim that detectors have 'low uncertainty' (Fig. 8) may therefore be a coverage artifact: they are confident because they only respond to clear faces, not because their representations are intrinsically certain. This selection bias also infects the GT-box control in §4.5, which reports only response rate and mean Human score, not RAI on the common GT-box set. Without computing a five-way distribution on the same regions for all models, the decoupling of uncertainty and bias (the paper's central finding) is not established. Additionally, the 'representation over thresholds' conclusion is asserted without a threshold sweep; LLaVA's near-deterministic RAI could simply be an artifact of prompt/parsing (also undisclosed).","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces face pareidolia as a diagnostic probe for vision models, proposing a unified protocol that measures detection, localization, uncertainty, and class bias on the FacesInThings dataset. It compares six off-the-shelf models across four regimes: contrastive VLMs (CLIP-B/32, CLIP-L/14), a generative VLM (LLaVA-1.5-7B), a pure vision classifier (ViT), a general detector (YOLOv8), and a face detector (RetinaFace). The central empirical claim is that uncertainty and bias are decoupled: low predictive entropy can signal either conservative suppression (detectors) or confident over-interpretation (VLMs), while high entropy can coexist with low directional bias (ViT). The paper further claims that behavior under ambiguity is governed more by representation than by score thresholds, and that VLMs systematically over-call non-human pareidolic regions as Human, with LLaVA being the most extreme.","tokens_in":9750,"tokens_out":3993,"duration_ms":44359,"significance":"If the central claim holds, the paper provides a valuable, compact diagnostic for evaluating semantic robustness and calibration under ambiguity, with implications for safety-critical vision systems. The choice of an external human-annotated dataset, the use of standard pretrained models without fine-tuning, and the inclusion of a ground-truth-box control for detectors are all strengths that make the result credible as an exploratory cross-regime comparison. However, the paper's headline conclusions currently rest on aggregate comparisons that lack error bars, significance tests, and exact numbers, and on an uncertainty metric whose computation is not matched across model regimes. These issues are fixable within the manuscript's scope, but they are load-bearing for the decoupling and representation-over-thresholds claims.","major_comments":[{"comment":"The Representation Ambiguity Index (RAI) is computed as an average entropy over matched regions (Eq. 4). For box-level models, all annotated regions are matched, so RAI averages over easy and hard cases alike. For detectors, a region enters the average only if the model produces a matched detection. Since YOLOv8 matches roughly 40% of primary regions and RetinaFace near 0% (Fig. 4), their low RAI reported in Fig. 8 may be a coverage artifact: the detector is confident simply because it only fires on its easiest, most face-like subset. This selection bias directly undermines the claim that detectors exhibit 'low uncertainty' as a representational property. The authors should compute a five-way distribution and RAI on the same GT-box regions for all models, or otherwise condition the RAI comparison on a common set of regions.","section":"§4.3, Eq. (4)"},{"comment":"The GT-box-controlled experiment is the right idea, but it is currently insufficient to support the uncertainty part of the decoupling claim. It reports only the binary response rate (Eq. 8) and mean Human score conditional on response (Eq. 9). It does not report RAI, the full five-class distribution, or any uncertainty measure on the GT-box crops. Therefore, the statement that detectors are 'confident and conservative' under controlled localization is not supported by the presented evidence. The control addresses localization bias but not the uncertainty comparison that is central to the paper's main finding. Please report uncertainty metrics on the common GT-box set for detectors and, ideally, for all models.","section":"§4.5"},{"comment":"The conclusion that behavior under ambiguity is 'governed more by representation than thresholds' is asserted without any threshold manipulation. No experiment varies detection confidence thresholds, classification decision thresholds, or VLM decoding temperatures. The abstract's 73% over-call figure for LLaVA and the qualitative separation in Fig. 6 and Fig. 8 are not accompanied by exact numerical values, confidence intervals, or significance tests. Since the central claim is explicitly about thresholds not being the determining factor, a threshold sweep is required: for each model, show that over-call rates and uncertainty profiles are stable across a reasonable range of decision thresholds, or quantify how much threshold tuning can change the reported pattern. Without this, the representational explanation remains one of several plausible interpretations.","section":"§6 and abstract"},{"comment":"The comparability of the six models depends on undisclosed evaluation choices. The CLIP class prototypes are 'a small set of natural descriptions' but the prompts are not listed; the LLaVA classification prompt and the text-parsing scheme are not specified; and the YOLOv8-to-five-class mapping is described only by two examples. Different prompt sets or parsing rules could plausibly explain part of the VLM over-call gap, especially LLaVA's near-deterministic behavior. The authors should provide the full prompt set, the parsing instructions, the exact YOLO category mapping, and the code or clear pseudocode. Without this, the cross-model alignment to a common five-class space is not reproducible, and the claim that the observed differences are representation-driven rather than evaluation-artifact-driven is weakened.","section":"§3.2 and §4.2"}],"minor_comments":[{"comment":"The definition of RAI is inconsistent: Eq. (3)–(4) say 'average entropy across images,' but the text immediately before says 'we aggregate class probabilities across matched regions.' Please clarify whether the average is over all images or only images with a matched prediction, and state this explicitly for each model family.","section":"§3.4"},{"comment":"The abstract reports 'LLaVA over-calling on 73% of non-human pareidolic images,' but this number does not appear in Section 4. Please include exact values for all key claims (non-human→Human, Alien→Human, RAI, response rates) in the text or tables, with sample sizes and confidence intervals.","section":"Abstract and §4.2"},{"comment":"The GT-box control crops 'a padded region around the annotation,' but the padding amount is not specified. Since detector response rates are sensitive to the input crop size and context, please report the exact padding protocol and, ideally, a sensitivity check over padding amounts.","section":"§4.5"},{"comment":"YOLOv8 is described as 'trained on COCO-style data'; please give the specific pretrained checkpoint (e.g., YOLOv8m-coco) and the confidence threshold or default settings used for detection in the full-image evaluation.","section":"§3.2"},{"comment":"Several figures are qualitative point plots without error bars or confidence ellipses (e.g., Fig. 6, Fig. 8). Adding error bars or bootstrapped confidence regions would make the cross-model separation more convincing and easier to judge.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is a diagnostic study rather than a new method, so its contribution rests entirely on the reliability of its measurements and the strength of the empirical separation. The RAI coverage confound and the missing threshold sweep are the two issues I would prioritize; if the authors can compute uncertainty metrics on a common GT-box region set and show that the three-mechanism pattern survives reasonable evaluation choices, the paper would be suitable for publication. I do not see a fundamental flaw in the central idea, but the current evidence is too selective to support the strong threshold-free conclusion."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea here is worth taking seriously: use face pareidolia as a controlled diagnostic to see how different model families handle ambiguous inputs. The cross-regime comparison—CLIPs, LLaVA, ViT, YOLOv8, RetinaFace—under one protocol is genuinely new, and the three-way behavioral split (VLM overactivation, ViT diffuse abstention, detectors conservative suppression) is a plausible and useful organizing observation. The GT-box control in §4.5 is a good instinct: it separates localization failure from semantic gating, and the finding that detectors stay conservative even on oracle boxes is real evidence for representation-driven priors rather than mere missed detections.\n\nThe soft spots are, unfortunately, in the load-bearing part of the paper. The central claim that “uncertainty and bias are decoupled” rests on RAI, but RAI for detectors is computed only on regions where the detector actually produced a matched detection. YOLOv8 matches ~40% of primary regions and RetinaFace ~0%, so their RAI averages over a small, likely easy, face-like subset. Low entropy among detectors could just mean they only respond when confident—a coverage artifact, not a representational property. The GT-box control reports response rate and mean Human score, but not RAI on the same boxes for all models, so it doesn't fix the comparison. Without a five-way distribution computed on identical regions for every model, the decoupling claim isn't established. Also, the paper asserts “representation over thresholds” without ever running a threshold sweep; that conclusion is premature. And the prompts for CLIP and LLaVA, plus the parsing scheme, are undisclosed—these directly drive the headline bias numbers, so the results are not yet reproducible. No error bars or statistical tests anywhere, and the abstract’s “73%” figure doesn't appear with exact support in the text.\n\nNone of this is fatal: the flaws are fixable with a revised evaluation that computes a common-region RAI, discloses the prompt set, and adds basic uncertainty quantification. As it stands, I'd treat the specific numbers as preliminary, but the diagnostic framework is a worthwhile contribution to model evaluation. It deserves a serious referee—send it out, but expect a request for major revisions.\n\nI wouldn't cite it yet, and I'd bring it to a reading group mainly to discuss the selection-bias pitfall in uncertainty metrics.","headline":"A useful diagnostic framing with a genuinely interesting cross-model observation, but the paper's central uncertainty–bias decoupling claim is currently under-supported by the reported metrics.","tokens_in":10222,"tokens_out":1487,"would_cite":false,"duration_ms":17958,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Face pareidolia reveals three distinct ways vision models handle ambiguity — and shows that a model's confidence is not a reliable safety signal.","keywords":["face pareidolia","visual ambiguity","representation bias","uncertainty quantification","vision-language models","model diagnostics","semantic overactivation","hard negatives"],"falsifier":"If using a large set of randomly sampled natural-language prompts for the five classes (rather than the authors' undisclosed set) changes the rank order of Human over-call rates across CLIP and LLaVA, the claim that bias is representation-driven rather than prompt-driven would be weakened. Equivalently, if a VLM fine-tuned on pareidolia images as hard negatives still over-calls Human with the same confidence, the representation-prior explanation would fail.","tokens_in":9349,"feed_emoji":"👁","tokens_out":4083,"duration_ms":35418,"temperature":0.7,"pith_summary":"This paper argues that when visual evidence is ambiguous, how a model interprets face-like patterns is determined by the way it represents semantics, not by its decision thresholds. To show this, it uses face pareidolia—seeing faces in objects—as a controlled probe and compares six models from four families. It finds that uncertainty and bias are separate: a model can be confident and wrong (vision-language models over-calling 'human') or uncertain and safe (a pure vision transformer staying diffuse). The practical stakes: confidence scores cannot be read as safety indicators on ambiguous inputs, and fixing bias requires changing representations, not tuning thresholds.","feed_headline":"Pareidolia reveals why vision models over-call 'human' on ambiguous images","feed_subtitle":"Six models, four architectures: confidence and bias don't move together, so threshold tuning won't fix it.","key_machinery":"The diagnostic pipeline that carries the argument aligns all six models to a common five-class space (Human, Animal, Cartoon, Alien, Other) and computes a small set of interpretable metrics: detection rate vs. primary pareidolia detection rate (to separate coverage from localization), the Representation Ambiguity Index (Shannon entropy of the class distribution), the False Bias Score and non-human→Human rate (to measure directional bias), and a ground-truth-box-controlled evaluation that isolates semantic gating from localization. The one-to-one greedy matching rule (IoU≥0.2 or center inclusion) connects detectors' boxes to annotated regions, allowing all models to be compared on the same am","core_discovery":"The paper's central discovery is that face pareidolia reveals three distinct mechanisms of handling ambiguity across model families. Vision-language models, especially the generative LLaVA, exhibit semantic overactivation: they systematically assign 'Human' to non-human pareidolic regions, with LLaVA over-calling on 73% of such images and doing so with near-zero uncertainty. A pure vision transformer (ViT) instead spreads probability across classes—an uncertainty-as-abstention strategy that keeps bias low. Detection models (YOLOv8, RetinaFace) achieve low bias through conservative priors that suppress responses, even when localization is controlled by giving them ground-truth boxes. The key","pith_inferences":["The decoupling of uncertainty and bias likely generalizes beyond face pareidolia to other categories with weak evidence (e.g., figures in fog, medical shadows), so the diagnostic could be extended to other semantic domains.","The hand-picked prompts used to align CLIP and LLaVA into the five-class space are a likely confound; comparing across many prompt sets could show whether some of the observed 'representation' difference is actually prompt-sensitivity.","The framework implies that human interpretation—where we readily see faces in objects—may be closer to the VLM's over-activation than to the detectors' suppression, which raises a question about what 'correct' behavior under ambiguity should be.","A testable extension: training a detector with pareidolia examples as hard negatives should shift its conservative prior, predicting a measurable increase in Human response rate under GT-box control."],"forward_implications":["Model confidence scores should not be used as a safety signal on ambiguous face-like inputs: LLaVA is both most confident and most biased.","Mitigating the bias requires changing prompts, priors, or training data, not threshold tuning.","Pareidolic images can serve as structured hard negatives for training more ambiguity-aware vision and vision-language systems.","Scale and generative alignment do not remove the bias: the largest and most capable VLM in the study was also the most over-interpretive.","Emotional content modulates bias in VLMs, meaning affective cues can act as semantic evidence for 'human' and need to be accounted for in deployment."],"fun_headline_variants":["VLMs over-call 'human' on 73% of ambiguous images","Pareidolia probe splits vision models into three ambiguity strategies","Low confidence doesn't mean low bias in vision models","LLaVA's semantic overactivation: too quick to see faces","Representation, not thresholds, drives ambiguity handling"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The comparison assumes that the hand-chosen prompts, LLaVA's parsing scheme, and YOLO's category mappings align all six models into the same five-class decision space with equal difficulty, and that the IoU≥0.2 matching rule does not favor one family of models over another.","fun_headline_variants_meta":{"raw":{"variants":["VLMs over-call 'human' on 73% of ambiguous images","Pareidolia probe splits vision models into three ambiguity strategies","Low confidence doesn't mean low bias in vision models","LLaVA's semantic overactivation: too quick to see faces","Representation, not thresholds, drives ambiguity handling"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000198,"raw_usage":{"total_tokens":1226,"prompt_tokens":790,"completion_tokens":436,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":534,"completion_tokens_details":{"reasoning_tokens":361}},"tokens_in":534,"tokens_out":436,"duration_ms":4529,"temperature":1.0,"reasoning_tokens":361,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T18:56:11.403939+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"If using a large set of randomly sampled natural-language prompts for the five classes (rather than the authors' undisclosed set) changes the rank order of Human over-call rates across CLIP and LLaVA, the claim that bias is representation-driven rather than prompt-driven would be weakened. Equivalently, if a VLM fine-tuned on pareidolia images as hard negatives still over-calls Human with the same confidence, the representation-prior explanation would fail.","supporting_citations":[],"review_version":1}