Pith. sign in

REVIEW 4 major objections 5 minor 17 references

When Visual Evidence is Ambiguous: Pareidolia as a Diagnostic Probe for Vision Models

T0 review · 4 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash

Pith's one-line read Face pareidolia reveals three distinct ways vision models handle ambiguity — and shows that a model's confidence is not a reliable safety signal.

desk verdict A useful diagnostic framing with a genuinely interesting cross-model observation, but the paper's central uncertainty–bias decoupling claim is currently under-supported by the reported metrics. read the letter →

arxiv 2603.03989 v3 pith:3QLI34WS submitted 2026-03-04 cs.CV cs.AI

classification cs.CVcs.AI
keywords facepareidoliavisualambiguityrepresentationbiasuncertaintyquantificationvision-languagemodelsmodeldiagnosticssemanticoveractivationhardnegatives
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that when visual evidence is ambiguous, how a model interprets face-like patterns is determined by the way it represents semantics, not by its decision thresholds. To show this, it uses face pareidolia—seeing faces in objects—as a controlled probe and compares six models from four families. It finds that uncertainty and bias are separate: a model can be confident and wrong (vision-language models over-calling 'human') or uncertain and safe (a pure vision transformer staying diffuse). The practical stakes: confidence scores cannot be read as safety indicators on ambiguous inputs, and fixing bias requires changing representations, not tuning thresholds.

What carries the argument

The diagnostic pipeline that carries the argument aligns all six models to a common five-class space (Human, Animal, Cartoon, Alien, Other) and computes a small set of interpretable metrics: detection rate vs. primary pareidolia detection rate (to separate coverage from localization), the Representation Ambiguity Index (Shannon entropy of the class distribution), the False Bias Score and non-human→Human rate (to measure directional bias), and a ground-truth-box-controlled evaluation that isolates semantic gating from localization. The one-to-one greedy matching rule (IoU≥0.2 or center inclusion) connects detectors' boxes to annotated regions, allowing all models to be compared on the same am

What would settle it

If using a large set of randomly sampled natural-language prompts for the five classes (rather than the authors' undisclosed set) changes the rank order of Human over-call rates across CLIP and LLaVA, the claim that bias is representation-driven rather than prompt-driven would be weakened. Equivalently, if a VLM fine-tuned on pareidolia images as hard negatives still over-calls Human with the same confidence, the representation-prior explanation would fail.

Watch

Extended reading notes

Core claim

The paper's central discovery is that face pareidolia reveals three distinct mechanisms of handling ambiguity across model families. Vision-language models, especially the generative LLaVA, exhibit semantic overactivation: they systematically assign 'Human' to non-human pareidolic regions, with LLaVA over-calling on 73% of such images and doing so with near-zero uncertainty. A pure vision transformer (ViT) instead spreads probability across classes—an uncertainty-as-abstention strategy that keeps bias low. Detection models (YOLOv8, RetinaFace) achieve low bias through conservative priors that suppress responses, even when localization is controlled by giving them ground-truth boxes. The key

Load-bearing premise

The comparison assumes that the hand-chosen prompts, LLaVA's parsing scheme, and YOLO's category mappings align all six models into the same five-class decision space with equal difficulty, and that the IoU≥0.2 matching rule does not favor one family of models over another.

Editorial extensions

If this is right

  • Model confidence scores should not be used as a safety signal on ambiguous face-like inputs: LLaVA is both most confident and most biased.
  • Mitigating the bias requires changing prompts, priors, or training data, not threshold tuning.
  • Pareidolic images can serve as structured hard negatives for training more ambiguity-aware vision and vision-language systems.
  • Scale and generative alignment do not remove the bias: the largest and most capable VLM in the study was also the most over-interpretive.
  • Emotional content modulates bias in VLMs, meaning affective cues can act as semantic evidence for 'human' and need to be accounted for in deployment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The decoupling of uncertainty and bias likely generalizes beyond face pareidolia to other categories with weak evidence (e.g., figures in fog, medical shadows), so the diagnostic could be extended to other semantic domains.
  • The hand-picked prompts used to align CLIP and LLaVA into the five-class space are a likely confound; comparing across many prompt sets could show whether some of the observed 'representation' difference is actually prompt-sensitivity.
  • The framework implies that human interpretation—where we readily see faces in objects—may be closer to the VLM's over-activation than to the detectors' suppression, which raises a question about what 'correct' behavior under ambiguity should be.
  • A testable extension: training a detector with pareidolia examples as hard negatives should shift its conservative prior, predicting a measurable increase in Human response rate under GT-box control.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces face pareidolia as a diagnostic probe for vision models, proposing a unified protocol that measures detection, localization, uncertainty, and class bias on the FacesInThings dataset. It compares six off-the-shelf models across four regimes: contrastive VLMs (CLIP-B/32, CLIP-L/14), a generative VLM (LLaVA-1.5-7B), a pure vision classifier (ViT), a general detector (YOLOv8), and a face detector (RetinaFace). The central empirical claim is that uncertainty and bias are decoupled: low predictive entropy can signal either conservative suppression (detectors) or confident over-interpretation (VLMs), while high entropy can coexist with low directional bias (ViT). The paper further claims that behavior under ambiguity is governed more by representation than by score thresholds, and that VLMs systematically over-call non-human pareidolic regions as Human, with LLaVA being the most extreme.

Significance. If the central claim holds, the paper provides a valuable, compact diagnostic for evaluating semantic robustness and calibration under ambiguity, with implications for safety-critical vision systems. The choice of an external human-annotated dataset, the use of standard pretrained models without fine-tuning, and the inclusion of a ground-truth-box control for detectors are all strengths that make the result credible as an exploratory cross-regime comparison. However, the paper's headline conclusions currently rest on aggregate comparisons that lack error bars, significance tests, and exact numbers, and on an uncertainty metric whose computation is not matched across model regimes. These issues are fixable within the manuscript's scope, but they are load-bearing for the decoupling and representation-over-thresholds claims.

major comments (4)
  1. [§4.3, Eq. (4)] The Representation Ambiguity Index (RAI) is computed as an average entropy over matched regions (Eq. 4). For box-level models, all annotated regions are matched, so RAI averages over easy and hard cases alike. For detectors, a region enters the average only if the model produces a matched detection. Since YOLOv8 matches roughly 40% of primary regions and RetinaFace near 0% (Fig. 4), their low RAI reported in Fig. 8 may be a coverage artifact: the detector is confident simply because it only fires on its easiest, most face-like subset. This selection bias directly undermines the claim that detectors exhibit 'low uncertainty' as a representational property. The authors should compute a five-way distribution and RAI on the same GT-box regions for all models, or otherwise condition the RAI comparison on a common set of regions.
  2. [§4.5] The GT-box-controlled experiment is the right idea, but it is currently insufficient to support the uncertainty part of the decoupling claim. It reports only the binary response rate (Eq. 8) and mean Human score conditional on response (Eq. 9). It does not report RAI, the full five-class distribution, or any uncertainty measure on the GT-box crops. Therefore, the statement that detectors are 'confident and conservative' under controlled localization is not supported by the presented evidence. The control addresses localization bias but not the uncertainty comparison that is central to the paper's main finding. Please report uncertainty metrics on the common GT-box set for detectors and, ideally, for all models.
  3. [§6 and abstract] The conclusion that behavior under ambiguity is 'governed more by representation than thresholds' is asserted without any threshold manipulation. No experiment varies detection confidence thresholds, classification decision thresholds, or VLM decoding temperatures. The abstract's 73% over-call figure for LLaVA and the qualitative separation in Fig. 6 and Fig. 8 are not accompanied by exact numerical values, confidence intervals, or significance tests. Since the central claim is explicitly about thresholds not being the determining factor, a threshold sweep is required: for each model, show that over-call rates and uncertainty profiles are stable across a reasonable range of decision thresholds, or quantify how much threshold tuning can change the reported pattern. Without this, the representational explanation remains one of several plausible interpretations.
  4. [§3.2 and §4.2] The comparability of the six models depends on undisclosed evaluation choices. The CLIP class prototypes are 'a small set of natural descriptions' but the prompts are not listed; the LLaVA classification prompt and the text-parsing scheme are not specified; and the YOLOv8-to-five-class mapping is described only by two examples. Different prompt sets or parsing rules could plausibly explain part of the VLM over-call gap, especially LLaVA's near-deterministic behavior. The authors should provide the full prompt set, the parsing instructions, the exact YOLO category mapping, and the code or clear pseudocode. Without this, the cross-model alignment to a common five-class space is not reproducible, and the claim that the observed differences are representation-driven rather than evaluation-artifact-driven is weakened.
minor comments (5)
  1. [§3.4] The definition of RAI is inconsistent: Eq. (3)–(4) say 'average entropy across images,' but the text immediately before says 'we aggregate class probabilities across matched regions.' Please clarify whether the average is over all images or only images with a matched prediction, and state this explicitly for each model family.
  2. [Abstract and §4.2] The abstract reports 'LLaVA over-calling on 73% of non-human pareidolic images,' but this number does not appear in Section 4. Please include exact values for all key claims (non-human→Human, Alien→Human, RAI, response rates) in the text or tables, with sample sizes and confidence intervals.
  3. [§4.5] The GT-box control crops 'a padded region around the annotation,' but the padding amount is not specified. Since detector response rates are sensitive to the input crop size and context, please report the exact padding protocol and, ideally, a sensitivity check over padding amounts.
  4. [§3.2] YOLOv8 is described as 'trained on COCO-style data'; please give the specific pretrained checkpoint (e.g., YOLOv8m-coco) and the confidence threshold or default settings used for detection in the full-image evaluation.
  5. [General] Several figures are qualitative point plots without error bars or confidence ellipses (e.g., Fig. 6, Fig. 8). Adding error bars or bootstrapped confidence regions would make the cross-model separation more convincing and easier to judge.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: empirical benchmark on an external dataset, no fitted prediction or author-derived uniqueness claim.

full rationale

The paper does not derive its conclusions from assumptions that already contain them. It evaluates six pretrained checkpoints on FacesInThings, an external published dataset [5], and reports descriptive metrics: detection rate, PPDR, RAI, FBS, and GT-box response rates. No parameter is fitted to the target claim and then renamed as a prediction; no uniqueness theorem or ansatz is imported from the authors' own prior work; citations are to external sources. The central uncertainty–bias decoupling is an empirical observation, not a mathematical consequence of the metric definitions. The main legitimate concerns are construct-validity issues: for detectors, the five-way probability vector is built from a one-hot mapping, so RAI is near-zero partly by output representation, and RAI averages only over matched regions, which can create a coverage-selection confound; CLIP prompts and LLaVA parsing are also undisclosed. These are correctness/validity concerns, not circular reasoning under the stated criteria. The paper itself flags the detector RAI caveat ('Their near-zero entropy reflects hard gating driven by strong priors, not calibrated semantic uncertainty in the five-way space'), so the limitation is acknowledged rather than hidden. Accordingly, no circular step is present.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

No fitted parameters in the usual sense, but the evaluation is governed by hand-chosen thresholds, prompts, and label mappings. No new physical or architectural entities are introduced. The empirical claims rest on the domain assumptions above, especially the commensurability of the five-class space across very different model families.

free parameters (3)
  • IoU matching threshold = 0.2 (plus center-inclusion rule)
    Hand-chosen relaxed spatial criterion in §3.3; directly affects PPDR and matching, and thus detection rates by difficulty.
  • CLIP class prompt set = unspecified
    §3.2 says “a small set of natural descriptions” per coarse class; prompts are not listed, so Human-bias rates depend on unstated text choices.
  • GT-box crop padding = unspecified
    §4.5 says “crop a padded region around the annotation” but padding is not quantified; affects detector response rates in the control experiment.
assumptions (5)
  • domain assumption FacesInThings human annotations are reliable ground truth for presence and location of pareidolic regions, and difficulty/emotion labels are meaningful.
    Used as GT in all metrics; if annotations are noisy, bias rates are mis-calibrated. Invoked throughout §3.1–4.
  • domain assumption All model outputs can be aligned to the same five-class space without loss of the differences being studied.
    CLIP/ViT prototypes, LLaVA text responses, YOLO categories, and RetinaFace detections are mapped to {Human, Animal, Cartoon, Alien, Other}; if mapping distorts semantics, cross-model comparisons are artifacts. §3.2.
  • domain assumption For box-level classifiers, providing the annotated crop does not give away the human-face answer; the model still must decide among classes.
    Box-level models get perfect detection by construction; the paper relies on this to separate detection from bias, but cropping already removes localization ambiguity. §3.3, §4.1.
  • domain assumption Entropy over the five coarse classes is a valid measure of representational ambiguity.
    RAI is defined in §3.4(3); treating entropy as uncertainty assumes the five-way distribution captures the model's relevant uncertainty.
  • standard math Shannon entropy and conditional probability definitions behave as usual.
    Used in RAI and FBS; not contentious.

how reviews work

0 comments
Cite this review

Pith. "Pith review of When Visual Evidence is Ambiguous: Pareidolia as a Diagnostic Probe for Vision Models." pith.science (2026). https://pith.science/paper/3QLI34WS

@misc{pith2026260303989,
  author       = {Pith},
  title        = {Pith review of: When Visual Evidence is Ambiguous: Pareidolia as a Diagnostic Probe for Vision Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3QLI34WS}},
  note         = {Machine review of arXiv:2603.03989}
}
read the original abstract

When visual evidence is ambiguous, vision models must decide how to interpret face-like patterns. Face pareidolia, the perception of faces in non-face objects, provides a controlled probe of such decisions. We introduce a diagnostic framework that analyzes detection, localization, uncertainty and bias across class, difficulty and emotion. We evaluate six models spanning four representational regimes: vision-language models (VLMs; CLIP-B/32, CLIP-L/14, LLaVA-1.5-7B), pure vision classification (ViT), object detection (YOLOv8), and face detection (RetinaFace). Our results reveal that uncertainty and bias are decoupled: low uncertainty can signal either safe suppression, as in detectors, or extreme over-interpretation, as in VLMs. VLMs exhibit semantic overactivation, systematically interpreting ambiguous non-human regions as Human, with LLaVA over-calling on 73% of non-human pareidolic images, especially for negative emotions. ViT instead follows an uncertainty-as-abstention strategy, remaining diffuse yet largely unbiased. Detection-based models achieve low bias through conservative priors that suppress pareidolia responses even when localization is controlled. Together, these results show that behavior under ambiguity is governed more by representation than thresholds, establishing face pareidolia as a diagnostic of semantic robustness and a source of ambiguity-aware hard negatives for vision models. Code will be released upon publication.

Figures

Figures reproduced from arXiv: 2603.03989 by the authors.

Figure 1
Figure 1. Face pareidolia in an electrical outlet. The visual input is unchanged, yet observers may perceive a face, il￾lustrating how interpretation emerges under ambiguity. Hamilton et al. [5] introduced the FacesInThings dataset and examined face detectors on pareidolic stimuli, establish￾ing pareidolia as a challenge. We instead treat pareidolia as a representation-level probe, using the same stimuli to analyze model resp… view at source ↗
Figure 2
Figure 2. Example images from the FacesInThings dataset [5]. Red bounding boxes in￾dicate face-like regions perceived by human observers in otherwise inanimate objects. 3 Method Our approach combines a unified diagnostic pipeline with a compact set of inter￾pretable metrics, using pareidolic inputs not just as a benchmark but as a tool for characterizing how different models organize and express semantic evidence under ambigu… view at source ↗
Figure 3
Figure 3. Unified pareidolia diagnostic pipeline. VLMs (CLIP-B/32, CLIP-L/14, LLaVA￾1.5-7B) and pure vision classifier (ViT) classify annotated regions, while general object detection (YOLOv8) and face-specific detection (RetinaFace) detect faces in full im￾ages. Predictions are mapped to a common five-class space for evaluation of detection, localization, uncertainty and bias across subgroups. 4 Experiments and Results 4.1 D… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Detection coverage vs. localiza￾tion. Box-level classifiers saturate both metrics, while YOLOv8 and RetinaFace under-respond [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 6
Figure 6. Figure 6: Human over-call bias on pareidolic regions. Points show False Bias Score (FBS) versus non-human→Human rate, revealing stronger bias in VLMs (espe￾cially LLaVA) than in ViT, YOLOv8, and RetinaFace [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 8
Figure 8. Figure 8: Prediction uncertainty (RAI) on pareidolic regions. ViT is uncertain and un￾biased, CLIP moderately uncertain and biased, LLaVA confident and strongly biased, detectors confident and conservative [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 9
Figure 9. Figure 9: Non-human→Human bias by emotion. VLMs show the highest over-call rates, with CLIP especially biased on neg￾ative emotions. ViT, YOLOv8 and Reti￾naFace remain near zero [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]
Figure 11
Figure 11. Figure 11: GT-box-controlled detector behavior on pareidolic regions across difficulty levels. Left: response rate (probability of any Human detection on a GT box). Right: mean Human score conditional on responding [PITH_FULL_IMAGE:figures/full_fig_p012_11.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

17 extracted references · 2 linked inside Pith

  1. [1]

    British Journal of Psychology113(2), 496–507 (2022)

    Caruana, N., Seymour, K.: Objects that induce face pareidolia are prioritized by the visual system. British Journal of Psychology113(2), 496–507 (2022)

  2. [2]

    ACM trans- actions on Graphics (TOG)42(4), 1–10 (2023)

    Chefer, H., Alaluf, Y., Vinker, Y., Wolf, L., Cohen-Or, D.: Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models. ACM trans- actions on Graphics (TOG)42(4), 1–10 (2023)

  3. [3]

    In: Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition

    Deng, J., Guo, J., Ververas, E., Kotsia, I., Zafeiriou, S.: Retinaface: Single-shot multi-level face localisation in the wild. In: Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition. pp. 5203–5212 (2020)

  4. [4]

    arXiv preprint arXiv:2010.11929 (2020)

    Dosovitskiy, A., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)

  5. [5]

    In: European conference on computer vision

    Hamilton, M., Stent, S., DuTell, V., Harrington, A., Corbett, J., Rosenholtz, R., Freeman, W.T.: Seeing faces in things: A model and dataset for pareidolia. In: European conference on computer vision. pp. 377–395. Springer (2024)

  6. [6]

    IEEE Transactions on Pattern Analysis and Machine Intelligence47(5), 3563–3579 (2025)

    Huang, K., Duan, C., Sun, K., Xie, E., Li, Z., Liu, X.: T2I-CompBench++: An en- hanced and comprehensive benchmark for compositional text-to-image generation. IEEE Transactions on Pattern Analysis and Machine Intelligence47(5), 3563–3579 (2025)

  7. [7]

    Procedia computer science199, 1066–1073 (2022)

    Jiang, P., Ergu, D., Liu, F., Cai, Y., Ma, B.: A review of yolo algorithm develop- ments. Procedia computer science199, 1066–1073 (2022)

  8. [8]

    arXiv preprint arXiv:2309.14381 (2023)

    Lee, N., Bang, Y., Lovenia, H., Cahyawijaya, S., Dai, W., Fung, P.: Survey of social bias in vision-language models. arXiv preprint arXiv:2309.14381 (2023)

Show all 17 references
  1. [9]

    Advances in Neural Information Processing Systems36, 69981–70011 (2023)

    Lee, T., et al.: Holistic evaluation of text-to-image models. Advances in Neural Information Processing Systems36, 69981–70011 (2023)

  2. [10]

    Advances in neural information processing systems36, 34892–34916 (2023)

    Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. Advances in neural information processing systems36, 34892–34916 (2023)

  3. [11]

    Cortex53, 60–77 (2014)

    Liu, J., Li, J., Feng, L., Li, L., Tian, J., Lee, K.: Seeing jesus in toast: neural and behavioral correlates of face pareidolia. Cortex53, 60–77 (2014)

  4. [12]

    International journal of com- puter vision128(2), 261–318 (2020)

    Liu, L., Ouyang, W., Wang, X., Fieguth, P., Chen, J., Liu, X., Pietikäinen, M.: Deep learning for generic object detection: A survey. International journal of com- puter vision128(2), 261–318 (2020)

  5. [13]

    In: International conference on machine learning

    Radford, A., et al.: Learning transferable visual models from natural language su- pervision. In: International conference on machine learning. pp. 8748–8763. PmLR (2021)

  6. [14]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: Unified, real-time object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 779–788 (2016)

  7. [15]

    Interna- tional journal of computer vision115(3), 211–252 (2015)

    Russakovsky, O., et al.: Imagenet large scale visual recognition challenge. Interna- tional journal of computer vision115(3), 211–252 (2015)

  8. [16]

    Zhou, K., Lai, E., Jiang, J.: Vlstereoset: A study of stereotypical bias in pre- trained vision-language models. In: Proceedings of the 2nd Conference of the Asia- Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on N...

  9. [17]

    Journal of Pacific Rim Psychology14, e2 (2020)

    Zhou, L.F., Meng, M.: Do you see the “face”? individual differences in face parei- dolia. Journal of Pacific Rim Psychology14, e2 (2020)

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.