Pith. sign in

REVIEW 3 major objections 6 minor 59 references

VisionTrap: Unanswerable Questions On Visual Data

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read State-of-the-art vision-language models answer questions that have no right answer, and offering choices makes it worse.

desk verdict VisionTrap builds a useful unanswerable-VQA benchmark, but the headline with/without-options contrast is confounded by the prompt format and needs re-running before the paper's central claim can stand. read the letter →

arxiv 2507.17262 v1 pith:RNKDZ4UV submitted 2025-07-23 cs.CV

Asir Saadat , Syem Aziz , Shahriar Mahmud , Abdullah Ibne Masud Mahi , Sabbir Ahmed This is my paper · ORCID
classification cs.CV
keywords visualquestionansweringabstentionunanswerablequestionsvision-languagemodelshallucinationmultiple-choicebiaszero-shotevaluationsyntheticimages
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that current vision-language models (VLMs) are biased toward producing answers even when a visual question has no correct answer, and that multiple-choice options amplify this bias. To test this, the authors built VisionTrap, a dataset of 300 impossible images with 1,500 logically unanswerable questions spanning surreal scenes, animal-object fusions, and existing fictional figures. They asked four models—LLaVA 7B, GPT-4o, GPT-4.1, and Gemini Flash 2.5—to answer either from four choices or freely, with an explicit reserved token for 'I cannot answer.' The central result is that abstention rates rise sharply when options are removed: GPT-4o, for example, goes from 0.571 to 0.892 abstention on the existing-figures category. The authors read this as evidence that models overapply learned answer patterns instead of recognizing the boundary of what an image or question can support.

What carries the argument

The load-bearing machinery is the VisionTrap benchmark together with a two-condition prompt protocol and a reserved abstention token. VisionTrap supplies 300 images that cannot occur in reality, each with five questions and four answer choices engineered to have no ground truth, organized into three image categories (surreal, fusion of objects and animals, existing figures) and five question subtypes per category. In the with-options condition the model picks among the four choices plus a fifth token, '5,' which is defined as abstention; in the without-options condition the same prompt asks the model to return '5' when its answer belongs to an 'uncertain set' of phrases such as 'The answer is unknown' and 'There is no definitive answer.' Abstention is quantified as $AR = (TN + FN)/|D|$ using a confusion matrix for answerable versus unanswerable questions, and the same protocol is run on answerable VQA v2 questions to detect false abstentions. The token '5' is the key operational object: every headline number in the paper is the rate at which a model emits that token.

What would settle it

Run the VisionTrap prompts on a matched set of answerable images, or reverse the wording and placement of the abstention instruction in the no-options condition; if abstention on unanswerable questions changes sharply with the wording, or models abstain on answerable questions at similar rates, the claimed recognition of unanswerability is not supported. The paper's own LLaVA result—about 97 percent abstention on answerable VQA v2 questions in the no-options condition—is already close to such a falsifying observation for that model.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that state-of-the-art VLMs frequently fail to abstain on unanswerable visual questions, and the failure is strongest when the model is given multiple-choice options. Across all three image categories and all four models, abstention was lower with options than without. The sharpest contrast is LLaVA 7B, which with options abstains on 0 to 4 percent of questions but without options abstains on 95 to 98 percent; the authors attribute the difference to the presence of distractor choices rather than to an underlying change in visual understanding. GPT-4o shows the most consistent abstention, exceeding 90 percent without options on two of three categories, while Gemini Flash 2.5 abstains least overall. The paper also probes the opposite failure: models abstaining on questions that do have answers. On a 1,000-question subset of VQA v2, the no-options prompt produces high false abstention, especially for LLaVA, which abstains on about 97 percent of answerable questions. The authors therefore frame their main result cautiously: the observed abstention rates measure a tendency shaped by question format, not a clean, calibrated awareness of ignorance.

Load-bearing premise

The paper's numbers all treat the token '5' as proof that a model recognized the question as unanswerable; if models are instead reacting to prompt wording, option position, or a learned default, the abstention rates measure instruction compliance rather than awareness of ignorance.

Editorial extensions

If this is right

  • Multiple-choice answer formats, which dominate current VQA benchmarks, systematically suppress abstention and push models to guess on questions with no right answer.
  • Benchmarks that measure only accuracy on answerable questions fail to expose a failure mode that matters in real deployments, where users may ask impossible or ill-posed questions.
  • Model rankings on 'honesty' are format-dependent: the same model can look near-random with options and strongly abstention-prone without them.
  • Training with unanswerable visual questions and explicit abstention supervision would be needed to close the gap, since current models are not reliably learning to say 'I don't know' from standard VQA data.
  • The false-abstention results imply that simply prompting for caution is not a fix: it trades hallucinations for refusal on answerable questions, especially for smaller open models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next benchmark would pair VisionTrap's unanswerable images with matched answerable control questions so that a single discrimination score, such as $d'$, separates genuine recognition of unanswerability from a blanket tendency to say '5'.
  • The paper's answer-distribution plots suggest a positional option-1 bias, which implies a cheap test: randomize option order and see whether abstention and option-1 rates move; if they do, part of the measured effect is interface bias rather than visual reasoning.
  • If the option-effect generalizes beyond these four models, product design becomes a lever for honesty: open-ended answer fields or an explicit 'insufficient information' button could reduce confident wrong answers in deployed assistants.
  • Because the dataset's images are synthetic or public-domain, the same protocol could be extended to video or interactive embodied settings, where unanswerable questions about impossible objects are more natural.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces VisionTrap, a dataset of 1,500 unanswerable visual questions across three image categories (surreal images, animal-object fusions, and existing fictional figures), and evaluates four VLMs—LLaVA 7B, GPT-4o, GPT-4.1, and Gemini Flash 2.5—under two prompt conditions: one with four multiple-choice options plus an abstention token ('5'), and one without options but still instructing the model to output an option number and to use '5' for uncertainty. Abstention rate is measured as the frequency of outputting the token '5'. The paper reports that models frequently fail to abstain when options are present, that removing options increases abstention substantially (e.g., GPT-4o from 0.571 to 0.892 on Existing images), and that models produce confident justifications for unanswerable questions. It also includes a false-negative analysis on answerable VQA v2 questions under the same prompt.

Significance. The dataset and the descriptive finding that commercial and open VLMs rarely abstain even when the prompt explicitly offers an abstention token are potentially useful for the VQA reliability community. The inclusion of a false-negative control (Section 4.3) and a justification taxonomy (Section 4.5) are constructive steps toward measuring abstention more carefully. However, the central claim about the effect of removing options is undermined by a confounded prompt design and by the paper's own false-negative results; the no-options abstention rates cannot currently be distinguished from instruction following. The contribution would be strengthened if the abstention signal were elicited in a way not tied to a single token and if the option-removal contrast were validated with calibrated metrics. The paper would also benefit from reporting uncertainty estimates; all numbers are point estimates with no confidence intervals or significance tests.

major comments (3)
  1. [Sec. 3.2, Table 1; Sec. 4.2, Table 2] The no-options condition in Table 1 still requires the model to output an option number ('Answer: <just write the option number>') and defines '5' as the uncertainty signal; because no options 1–4 are listed, '5' is the only format-compliant numeric response. The increase in abstention rates from the with-options to the without-options condition in Table 2 (e.g., GPT-4o from 0.571 to 0.892 on Existing) therefore cannot be attributed to the model recognizing unanswerability. This interpretation is directly contradicted by the paper's own Section 4.3/Fig. 6, where LLaVA's false-negative rate on answerable VQA v2 questions under the same no-options prompt is 0.972, showing that the prompt elicits '5' responses regardless of actual answerability.
  2. [Sec. 4.3, Fig. 6] The paper reports that LLaVA labels 97.2% of clearly answerable VQA v2 questions as unanswerable under the no-options prompt, and states that 'relying solely on the abstention rate from unanswerable questions may not accurately reflect the model's overall performance.' This concession is in direct tension with the use of the same uncalibrated abstention rates in Section 4.2 as evidence that 'models are more likely to correctly recognize unanswerable scenarios' when options are removed. The abstention metric should be calibrated against the false-negative rate, for example by reporting abstention behavior only on the subset of unanswerable questions for which the model also correctly answers answerable controls.
  3. [Sec. 4.6, Fig. 12] The with-options condition is itself contaminated by positional bias: LLaVA selects option 1 in about 99% of cases (Section 4.6), and GPT-4o and Gemini also skew strongly toward option 1 (Fig. 12). Consequently, the with-options abstention rates in Table 2 conflate genuine abstention decisions with a positional answer heuristic, and the with-options versus without-options comparison does not isolate the effect of providing options. The authors should randomize option order across items or use a free-form answer format to measure abstention without this confound.
minor comments (6)
  1. [Table 2] Report confidence intervals or standard errors. With 500 questions per category, the differences among models on the same condition (e.g., Gemini 0.61 versus GPT-4.1 0.792 on Existing without options) may be within sampling variability; the present point estimates do not support fine-grained ranking claims.
  2. [Eq. (1)] The definition AR = (TN + FN)/|D| is confusing for the VisionTrap dataset, which contains only unanswerable questions and therefore no false negatives; for that dataset the metric should be defined as TN/|D|, and the role of FN should be clarified.
  3. [Table 1] In the 'Prompt with Options' row, the option list reads 'option1, option2, option3, option1' but should presumably be 'option1, option2, option3, option4'; as printed, the list contains a duplicate and no fourth option.
  4. [Figure 4] The y-axis is labeled 'Accuracy' while the caption and text describe abstention rate; relabel the axis to 'Abstention Rate' to avoid confusion.
  5. [Section 6.3] The text mentions 'GPT-4.0' in the discussion of Fig. 12, although the evaluated model is GPT-4.1; correct the typo.
  6. [General] The paper does not state whether the VisionTrap dataset and evaluation code will be made publicly available; an availability statement would be important for a benchmark paper.

Circularity Check

1 steps flagged · score 4.0 of 10

The headline option-removal effect is built into the response format: with no options, '5' is the only valid option number, so the measured abstention jump reflects instruction compliance, not unanswerability recognition.

  1. self definitional [Section 3.2, Table 1 (Prompt without Options); Section 3.3, Eq. (1); Section 4.2]
    "Prompt without Options: 'Answer: <just write the option number>' ... 'If you think your answer aligns with uncertain answers, then return 5.' ... 'We designated the numeral ‘5’ as the abstention marker' and defined AR = X/|D|, X ∈ {TN, FN}."

    In the no-options condition, no options 1–4 exist, yet the required answer format still asks for an option number and defines '5' as the uncertainty token. Hence '5' is the only numeric response satisfying the format, so the Abstention Rate in that column equals by construction the rate of emitting the single compliant token. The with-options condition supplies competing tokens 1–4, so the Table 2 contrast (e.g., GPT-4o 0.571→0.892) is driven by the response-format change rather than by recognition of unanswerability. The paper's own Section 4.3 check confirms this: under the same no-options prompt, LLaVA labels 97.2% of clearly answerable VQA v2 questions as unanswerable (FN = 0.972), showing the prompt induces '5' responses independently of answerability.

full rationale

Most of the paper is an empirical benchmark, not a derivation, and I find no load-bearing self-citation or fitted-parameter-as-prediction circularity. The with-options abstention measurements (Table 2) and the VQA v2 false-negative check (Section 4.3) are independent behavioral evidence and keep the paper from collapsing entirely. The central option-removal claim, however, is partially circular: both Table 1 prompts retain the instruction to answer with an option number and define '5' as the abstention marker, so in the no-options condition '5' is the only syntactically valid numeric output. The large abstention increases in Table 2 therefore measure format compliance by construction; the paper itself concedes in Section 4.3 that 'relying solely on the abstention rate from unanswerable questions may not accurately reflect the model’s overall performance' and reports LLaVA's 0.972 false-negative rate under this prompt. I score 4 because the result is not forced by a self-citation chain, but the headline contrast reduces, in part, to the measurement instrument.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The paper introduces no theoretical entities. Its empirical measurement rests on the hand-selected abstention token '5' and on the assumptions that the questions are truly unanswerable, that abstention is the correct response, and that a single run per model is representative.

free parameters (1)
  • abstention marker token = 5
    The numeral '5' was chosen by the authors as the abstention signal based on preliminary observations (Section 3.3). All abstention rates are defined by this token, so it is a hand-selected parameter that directly determines the central measurements.
assumptions (3)
  • domain assumption The 1,500 questions in VisionTrap are genuinely unanswerable, i.e., no valid ground-truth answer exists from any source.
    The dataset's construction relies on the authors' judgment plus a second annotator (Section 3.1); no inter-annotator agreement or external verification is reported.
  • domain assumption Abstention is the correct behavior for all VisionTrap questions, and the abstention token '5' faithfully encodes a model's recognition of unanswerability.
    This assumption underlies the Abstention Rate metric (Eq. 1) and all conclusions; LLaVA's inconsistent use of '5' shows the token is not a faithful measure for all models.
  • domain assumption A single API run per model is representative of model behavior.
    No repeated sampling, temperature settings, or variance estimates are reported, yet differences of a few percentage points between models are discussed as meaningful (Sections 4.1-4.4).

how reviews work

0 comments
Cite this review

Pith. "Pith review of VisionTrap: Unanswerable Questions On Visual Data." pith.science (2026). https://pith.science/paper/RNKDZ4UV

@misc{pith2026250717262,
  author       = {Pith},
  title        = {Pith review of: VisionTrap: Unanswerable Questions On Visual Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RNKDZ4UV}},
  note         = {Machine review of arXiv:2507.17262}
}
read the original abstract

Visual Question Answering (VQA) has been a widely studied topic, with extensive research focusing on how VLMs respond to answerable questions based on real-world images. However, there has been limited exploration of how these models handle unanswerable questions, particularly in cases where they should abstain from providing a response. This research investigates VQA performance on unrealistically generated images or asking unanswerable questions, assessing whether models recognize the limitations of their knowledge or attempt to generate incorrect answers. We introduced a dataset, VisionTrap, comprising three categories of unanswerable questions across diverse image types: (1) hybrid entities that fuse objects and animals, (2) objects depicted in unconventional or impossible scenarios, and (3) fictional or non-existent figures. The questions posed are logically structured yet inherently unanswerable, testing whether models can correctly recognize their limitations. Our findings highlight the importance of incorporating such questions into VQA benchmarks to evaluate whether models tend to answer, even when they should abstain.

Figures

Figures reproduced from arXiv: 2507.17262 by the authors.

Figure 1
Figure 1. Sample image from the curated dataset showing a fusion [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Illustrative examples of unanswerable visual questions across three image categories. (Left) Surreal images with unnatural object [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Overview of the pipeline for generating and evaluating [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (8 more)
Figure 5
Figure 5. Figure 5: Confusion matrix that demonstrates True Positive, False [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Abstention rates of different large language models [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Performance comparison across different categories of illogical visual inputs, with and without answer options. Each radar [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Sunburst charts showing the distribution of question openers across three image categories: Fusion of Objects and Animals, [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 9
Figure 9. Figure 9: Example from the Surreal Images category with question types, where the image is intentionally unrealistic or dreamlike, often defying physical or logical laws [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]
Figure 10
Figure 10. Figure 10: Example from the Fusion of Objects and Animals category, showing an image and corresponding unanswerable visual questions. 3. Logic and Physics Violations: This category includes scenarios that violate established physical laws or log￾12 [PITH_FULL_IMAGE:figures/full…
Figure 11
Figure 11. Figure 11: Example from the Existing Figures category, featuring a real-world scene paired with deliberately unanswerable questions. Option 1 Option 2 Option 4 Option 3 Option 5 50 100 150 200 250 300 350 400 450 500 Existing Images GPT-4o GPT-4.1 Gemini 2.5 Flash LLaVA 7b Optio…
Figure 12
Figure 12. Figure 12: Answer distribution across multiple-choice options for four different model architectures over three question categories. Option [PITH_FULL_IMAGE:figures/full_fig_p013_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

59 extracted references · 45 canonical work pages

  1. [1]

    Reassessing evaluation practices in visual ques- tion answering: A case study on out-of-distribution general- ization

    Aishwarya Agrawal, Ivana Kajic, Emanuele Bugliarello, El- naz Davoodi, Anita Gergely, Phil Blunsom, and Aida Ne- matzadeh. Reassessing evaluation practices in visual ques- tion answering: A case study on out-of-distribution general- ization. In Andreas Vlachos and Isabelle Augenstein, edi- tors, Findings of the Association for Computational Linguis- tics:...

    work page 2023

  2. [2]

    Vqa: Visual question answering

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision , pages 2425– 2433, 2015. 1, 2

    work page 2015

  3. [3]

    Ffhq-uv: Normalized facial uv-texture dataset for 3d face reconstruction

    Haoran Bai, Di Kang, Haoxian Zhang, Jinshan Pan, and Lin- chao Bao. Ffhq-uv: Normalized facial uv-texture dataset for 3d face reconstruction. InProceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 362–371, 2023. 2

    work page 2023

  4. [4]

    Paligemma: A versatile 3b vlm for trans- fer, 2024

    Lucas Beyer, Andreas Steiner, Andr ´e Susano Pinto, Alexan- der Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Grycner, Alexey Gritsenko, Neil Houlsby, Manoj Kumar, Keran Rong, Julian Eisensch- los, Rishabh Kabra, Matthi...

    work page 2024

  5. [5]

    Imaginet: A multi- content dataset for generalizable synthetic image detection via contrastive learning

    Delyan Boychev and Radostin Cholakov. Imaginet: A multi- content dataset for generalizable synthetic image detection via contrastive learning. arXiv preprint arXiv:2407.20020 ,

    arXiv

  6. [6]

    Large scale gan training for high fidelity natural image synthesis,

    Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis,

    work page

  7. [7]

    Gemini 1.5 and flash 2.5 models, 2024

    Google DeepMind. Gemini 1.5 and flash 2.5 models, 2024. 4, 15

    work page 2024

  8. [8]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. 2

    2009

Show all 59 references
  1. [9]

    Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio

    Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks, 2014. 2

    2014

  2. [10]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Ba- tra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913...

    2017

  3. [11]

    Unanswerable visual question answer- ing

    Yanyang Guo, Fangkai Jiao, Zhiqi Shen, Liqiang Nie, and Mohan Kankanhalli. Unanswerable visual question answer- ing. arXiv preprint arXiv:2310.10942, 2023. 2

    2023 arXiv

  4. [12]

    Survey of visual question answering: Datasets and techniques

    Akshay Kumar Gupta. Survey of visual question answering: Datasets and techniques. arXiv preprint arXiv:1705.03865,

    arXiv

  5. [13]

    Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P

    Danna Gurari, Qing Li, Abigale J. Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P. Bigham. Vizwiz grand challenge: Answering visual questions from blind people, 2018. 2

    2018

  6. [14]

    Hudson and Christopher D

    Drew A. Hudson and Christopher D. Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering, 2019. 2

    2019

  7. [15]

    Lawrence Zitnick, and Ross Girshick

    Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elemen- tary visual reasoning, 2016. 2

    2016

  8. [16]

    An analysis of visual question answering algorithms

    Kushal Kafle and Christopher Kanan. An analysis of visual question answering algorithms. In Proceedings of the IEEE international conference on computer vision , pages 1965– 1973, 2017. 1

    1965

  9. [17]

    A style-based generator architecture for generative adversarial networks,

    Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks,

  10. [18]

    Auto-encoding varia- tional bayes, 2022

    Diederik P Kingma and Max Welling. Auto-encoding varia- tional bayes, 2022. 2

    2022

  11. [19]

    Camera-to-robot pose estimation from a single image

    Timothy E Lee, Jonathan Tremblay, Thang To, Jia Cheng, Terry Mosier, Oliver Kroemer, Dieter Fox, and Stan Birch- field. Camera-to-robot pose estimation from a single image. In 2020 IEEE International Conference on Robotics and Au- tomation (ICRA), pages 9426–9432. IEEE, 2020. 2

    2020

  12. [20]

    Visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023. 4

    2023

  13. [21]

    Do llms know when to not answer? investigating abstention abilities of large lan- guage models

    Nishanth Madhusudhan, Sathwik Tejaswi Madhusudhan, Vikas Yadav, and Masoud Hashemi. Do llms know when to not answer? investigating abstention abilities of large lan- guage models. arXiv preprint arXiv:2407.16221, 2024. 2, 4

    2024 arXiv

  14. [22]

    Ok-vqa: A visual question answering benchmark requiring external knowledge, 2019

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge, 2019. 2

    2019

  15. [23]

    Gandiffface: Controllable generation of synthetic datasets for face recognition with realistic varia- tions

    Pietro Melzi, Christian Rathgeb, Ruben Tolosana, Ruben Vera-Rodriguez, Dominik Lawatsch, Florian Domin, and Maxim Schaubert. Gandiffface: Controllable generation of synthetic datasets for face recognition with realistic varia- tions. In Proceedings of the IEEE/CVF Internationa...

    2023

  16. [24]

    Microsoft copilot designer

    Microsoft. Microsoft copilot designer. https://www. microsoft . com / en - us / microsoft - designer,

  17. [25]

    Chatgpt (mar 14 version) [large language model],

    OpenAI. Chatgpt (mar 14 version) [large language model],

  18. [26]

    Chatgpt with image capabilities

    OpenAI. Chatgpt with image capabilities. https:// openai.com/chatgpt , 2024. Accessed: 2025-05-04. 3, 4

    2024

  19. [27]

    Interpretable visual question answer- ing via reasoning supervision

    Maria Parelli, Dimitrios Mallis, Markos Diomataris, and Vassilis Pitsikalis. Interpretable visual question answer- ing via reasoning supervision. In 2023 IEEE International Conference on Image Processing (ICIP), pages 2525–2529. IEEE, 2023. 1 9

    2023

  20. [28]

    Attention sorting combats recency bias in long context language models, 2023

    Alexander Peysakhovich and Adam Lerer. Attention sorting combats recency bias in long context language models, 2023. 8

    2023

  21. [29]

    Unrealgt: using unreal engine to generate ground truth datasets

    Thomas Pollok, Lorenz Junglas, Boitumelo Ruf, and Arne Schumann. Unrealgt: using unreal engine to generate ground truth datasets. In Advances in Visual Computing: 14th Inter- national Symposium on Visual Computing, ISVC 2019, Lake Tahoe, NV , USA, October 7–9, 2019, Proceeding...

    2019

  22. [30]

    Zero-shot text-to-image generation

    Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea V oss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International confer- ence on machine learning, pages 8821–8831. Pmlr, 2021. 2

    2021

  23. [31]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 2

    2022

  24. [32]

    Towards vqa models that can read, 2019

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read, 2019. 2

    2019

  25. [33]

    Dare: Diverse visual question answering with robustness evaluation, 2024

    Hannah Sterz, Jonas Pfeiffer, and Ivan Vuli ´c. Dare: Diverse visual question answering with robustness evaluation, 2024. 1

    2024

  26. [34]

    Benchmarking hallucination in large language models based on unanswerable math word problem

    Yuhong Sun, Zhangyue Yin, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Hui Zhao. Benchmarking hallucination in large language models based on unanswerable math word problem. arXiv preprint arXiv:2403.03558, 2024. 2, 4

    2024 arXiv

  27. [35]

    Clip-up: Clip-based unanswerable problem detection for visual question answer- ing

    Ben Vardi, Oron Nir, and Ariel Shamir. Clip-up: Clip-based unanswerable problem detection for visual question answer- ing. arXiv preprint arXiv:2501.01371, 2025. 2

    2025

  28. [36]

    Towards reasoning ability in scene text visual question answering

    Qingqing Wang, Liqiang Xiao, Yue Lu, Yaohui Jin, and Hao He. Towards reasoning ability in scene text visual question answering. In Proceedings of the 29th ACM International Conference on Multimedia, pages 2281–2289, 2021. 1

    2021

  29. [37]

    Character- izing llm abstention behavior in science qa with context per- turbations

    Bingbing Wen, Bill Howe, and Lucy Lu Wang. Character- izing llm abstention behavior in science qa with context per- turbations. arXiv preprint arXiv:2404.12452, 2024. 4

    2024 arXiv

  30. [38]

    Reliable visual question answering: Abstain rather than answer incorrectly

    Spencer Whitehead, Suzanne Petryk, Vedaad Shakib, Joseph Gonzalez, Trevor Darrell, Anna Rohrbach, and Marcus Rohrbach. Reliable visual question answering: Abstain rather than answer incorrectly. In European Conference on Computer Vision, pages 148–166. Springer, 2022. 2

    2022

  31. [39]

    Visual question answer- ing: A survey of methods and datasets

    Qi Wu, Damien Teney, Peng Wang, Chunhua Shen, Anthony Dick, and Anton Van Den Hengel. Visual question answer- ing: A survey of methods and datasets. Computer Vision and Image Understanding, 163:21–40, 2017. 1

    2017

  32. [40]

    Magic-vqa: Multimodal and grounded inference with commonsense knowledge for visual question answer- ing

    Shuo Yang, Siwen Luo, Soyeon Caren Han, and Eduard Hovy. Magic-vqa: Multimodal and grounded inference with commonsense knowledge for visual question answer- ing. arXiv preprint arXiv:2503.18491, 2025. 1

    2025 arXiv

  33. [41]

    P Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonza- lez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023. 14

    2023

  34. [42]

    Visual7w: Grounded question answering in images, 2016

    Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. Visual7w: Grounded question answering in images, 2016. 2

    2016

  35. [44]

    Question Types and Classification Our dataset consists of three categories of images and five types of questions in each category

    Appendix 6.1. Question Types and Classification Our dataset consists of three categories of images and five types of questions in each category. 6.1.1 Surreal Images These images depict scenarios that could not possibly hap- pen in the real world due to violations of how the p...

  36. [45]

    The questions focus on identifying the purpose or role of the object as depicted in the im- age, often assessing whether its practical utility aligns with the surrounding scenario

    Function Inquiry: This category evaluates the plau- sibility of an object’s intended use or function within a given context. The questions focus on identifying the purpose or role of the object as depicted in the im- age, often assessing whether its practical utility aligns wi...

  37. [46]

    The questions assess how the design and presence of these components contribute to the object’s practical usabil- ity in real-world scenarios

    Component Inquiry: This subtype focuses on miss- ing or distorted essential components of an object. The questions assess how the design and presence of these components contribute to the object’s practical usabil- ity in real-world scenarios

  38. [47]

    The questions focus on how structural stability is obtained, despite the fact that such configurations would not be viable in real-world scenarios

    Structural Stability Inquiry: This category exam- ines physically unfeasible structures, such as gravity- defying constructions or impossible geometries that contradict the principles of physical stability. The questions focus on how structural stability is obtained, despite t...

  39. [48]

    These questions challenge the model’s ability to reason about material suitability within a given context

    Material Compatibility Inquiry: This category of questions inquires about unrealistic material proper- ties, such as objects made from incompatible or con- tradictory substances. These questions challenge the model’s ability to reason about material suitability within a given context

  40. [49]

    It inquires whether sen- sory attributes—such as heat, texture, or sound—are logically consistent with real-world experiences

    Sensory Function Inquiry: This subtype challenges the model’s understanding of expected sensory experi- ences associated with objects. It inquires whether sen- sory attributes—such as heat, texture, or sound—are logically consistent with real-world experiences. 10 Figure 8. Su...

  41. [50]

    Consider whether anatomical changes preserve or disrupt essential bodily functions of ani- mals

    Anatomical Function Inquiry: This subtype exam- ines the viability of altered physiological features in hybrid forms. Consider whether anatomical changes preserve or disrupt essential bodily functions of ani- mals. Examines how changes in body structure still allow the animal ...

  42. [51]

    This involves as- sessing whether the digestive systems or metabolic processes of the animal can function properly given that it is fused with objects

    Dietary Compatibility Inquiry: This explores the logical consistency of feeding behaviors in mixed species forms, assessing whether dietary habits from both sources can feasible co-exist. This involves as- sessing whether the digestive systems or metabolic processes of the ani...

  43. [52]

    Questions are related to how the animal moves or maintains balance in daily life, since they are not in their usual anatomical struc- ture

    Mobility Inquiry: This addresses the challenges of locomotion that arise from the combination of anatom- ically incompatible elements. Questions are related to how the animal moves or maintains balance in daily life, since they are not in their usual anatomical struc- ture

  44. [53]

    Assesses how animals interact in their daily lives

    Communication Inquiry: This subtype questions the mechanisms of sound or signal production in hybrids, investigating whether communication methods remain coherent or become biologically implausible. Assesses how animals interact in their daily lives

  45. [54]

    The questions are related to how the animals survive in their inherent ecosystem

    Adaptation Inquiry: This focuses on the feasibility of environmental integration or survival traits, evaluating whether the hybrid could realistically survive in any natural habitat. The questions are related to how the animals survive in their inherent ecosystem. 6.1.3 Existi...

  46. [55]

    These questions create log- ical contradictions about the character’s identity, exis- tence, or consciousness

    Identity and Existence Paradoxes: This subtype explores contradictions in self-awareness or identity, such as a character questioning their own reality or continuity across versions. These questions create log- ical contradictions about the character’s identity, exis- tence, o...

  47. [56]

    Ques- tions include scenarios involving time travel, causal- ity paradoxes, or alternate versions of a character

    Time and Causality Loops: These involve paradoxes or alternate timelines, challenging the model to reason about events that disrupt chronological logic. Ques- tions include scenarios involving time travel, causal- ity paradoxes, or alternate versions of a character. For exampl...

  48. [57]

    Example from the Existing Figures category, featuring a real-world scene paired with deliberately unanswerable questions

    Logic and Physics Violations: This category includes scenarios that violate established physical laws or log- 12 Figure 11. Example from the Existing Figures category, featuring a real-world scene paired with deliberately unanswerable questions. Option 1 Option 2 Option 3Optio...

  49. [58]

    Reality and Fiction Blending: This category in- volves scenarios where fictional logic is mixed with real-world constraints. These questions challenge the model to reconcile imaginative or fantasy-based rules—such as magical powers, futuristic technolo- gies, or mythical setti...

  50. [59]

    How does it function?

    Ethical and Philosophical Dilemmas: This sub- type presents scenarios that explore morality, personal agency, and difficult choices. These questions often place characters in situations that test their ethical be- liefs, value systems, or sense of responsibility, raising deepe...

  51. [2023]

    Accessed: 2025-05-04. 3

    2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.