Pith. sign in

REVIEW 4 major objections 6 minor 14 references

The paper argues that current vision-language models overstate their reasoning ability, and that real-world visual illusions can serve as a joint test of perception and reasoning that exposes the gap.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 02:02 UTC pith:PQJCAJ43

load-bearing objection A genuinely new real-world illusion benchmark with a believable low-score headline, but the GPT-4o judge and missing human baseline undermine the comparative claims until fixed. the 4 major comments →

arxiv 2607.27747 v1 pith:PQJCAJ43 submitted 2026-07-30 cs.CL cs.AI

Can LVLMs Uncover the Truth Behind Visual Illusions? An Analysis of Perceptual and Reasoning Capabilities

Liangjie Zhao , Jiaqing Lyu , Kexin Tang , Zecheng Fang , Rong Yin , Yulan Hu , Da Li , Jianing Li This is my paper
classification cs.CL cs.AI
keywords visual illusionslarge vision-language modelsreasoning evaluationperceptionthinking modealignmentbenchmark constructionhallucination
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that visual illusions are a sharp diagnostic for whether large vision-language models (LVLMs) can genuinely reason about the visual world, and that by this test the reasoning abilities of current models are 'not as advanced as claimed.' To make the case, the authors built IllusionReasoning, a benchmark of 650 real-world illusion images with more than 3,000 question-answer pairs covering detection, description, and causal reasoning. Across closed- and open-source models, the best score is about 61%, well below what perception-focused benchmarks suggest. The paper also finds that explicit thinking mode does not consistently help, and that models often answer with safe, neutral content that sidesteps the illusion rather than resolving it.

Core claim

On the paper's own terms, the central discovery is that when perception and reasoning are tested jointly on real-world visual illusions, a wide range of LVLMs perform poorly and below their advertised cognitive level. The evidence is the IllusionReasoning score distribution, plus two behavioral findings: thinking mode helps on causal-reasoning questions but can hurt simple detection tasks by triggering over-thinking, and models frequently produce 'safe' responses that neither affirm the illusion nor state physical reality. The authors interpret this as an alignment problem—models are optimized to avoid error rather than to commit to a reading of ambiguous content—and argue the benchmark reve

What carries the argument

The load-bearing instrument is IllusionReasoning: 650 unedited, real-world optical-illusion images, manually verified and categorized into five causes (morphology, color and background, space, light and shadow, association), with over 3,000 QA pairs split among detection (binary), description (open-ended), and reasoning (cause-explanation) questions. The benchmark's force comes from two design choices: real-world images instead of classic synthetic illusions, which blocks memorization shortcuts, and a mix of binary and open-ended questions, which blocks random guessing. All answers are graded by an automated judge (GPT-4o) that the paper validates against human annotations on 200 sampled cas

Load-bearing premise

The entire ranking of models rests on trusting a single automated judge (GPT-4o) to grade free-form answers; the paper's own appendix shows a case where this judge accepts a response humans reject, so systematic leniency would change the conclusions.

What would settle it

Re-score the complete set of model outputs with strict human annotators (or a second independent judge) using the same reference answers; if the best model's accuracy moves from ~61% to above 75%, or if the rank order among models changes materially, the paper's claim that reasoning is 'not as advanced as claimed' is not supported by the current evidence.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Perception-only evaluations overestimate LVLM capability; scores on benchmarks such as object existence or simple VQA will not predict performance on questions that require resolving the conflict between appearance and physical reality.
  • Deploying thinking mode is not universally beneficial: for straightforward detection questions it can degrade accuracy, so reasoning effort must be matched to task difficulty.
  • Alignment training that rewards not making mistakes encourages 'safe' neutral answers on ambiguous content; IllusionReasoning can serve as a probe for this failure mode.
  • Model scale and architecture do not determine illusion reasoning; the paper's results suggest training data and alignment choices matter more than parameter count.
  • IllusionReasoning provides a discriminative test that separates models on perception and reasoning jointly, making it usable as a benchmark for future optimization.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If these results hold, applications that let LVLMs act on visual input without human oversight—such as robot navigation, autonomous driving, or safety-critical visual question answering—could inherit the same failure: a painted object might be treated as real, or a real object dismissed as an illusion. The paper's examples make this risk concrete but do not test downstream tasks.
  • The 30–60% neutral-response rate suggests a testable hypothesis about current alignment practices: models may be rewarded for low error rates on benchmark-style questions, which systematically favors vague answers on ambiguous inputs. One could test this by measuring whether the neutral rate changes across temperature, decoding strategy, or RLHF variants.
  • The decision to use only real-world images is itself an extension of the anti-illusion literature; a natural next step is to create paired synthetic versions of the same scenes to separate prior-driven reasoning failures from genuine perceptual ones. The paper does not do this paired comparison.
  • Because all scores flow through one automated judge, an important extension is to release the model outputs and run human evaluation; if the judge's leniency is skewed toward certain answer styles, the model ranking and the 'reasoning is weak' conclusion could change. This is our inference, not the paper's claim.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces IllusionReasoning, a benchmark of 650 real-world optical-illusion images with over 3,000 human-annotated QA pairs covering detection, description, and reasoning questions across five illusion categories. The authors evaluate 17 LVLMs (closed- and open-source, 1B–397B) using GPT-4o as an automated judge, reporting best average accuracy of 61.39% (Gemini-3.1-Pro) and best open-source accuracy of 57.28% (Qwen3.5-397B), well below ceiling. They conclude that LVLMs' reasoning capabilities are 'not as advanced as claimed,' that thinking mode does not consistently help (and can hurt on detection, description, and multi-choice tasks), and that models often produce neutral 'safe' responses. The paper positions IllusionReasoning as a joint perception-and-reasoning diagnostic that is more discriminative than existing illusion benchmarks and less susceptible to memorization because it uses real-world images.

Significance. If the central claim is upheld, IllusionReasoning would be a valuable addition to the LVLM evaluation landscape: it targets joint perception and reasoning, uses real-world images to reduce memorization of canonical synthetic illusions, and combines binary and open-ended questions. The annotation pipeline with multiple annotators and cross-verification is a genuine strength, as is the paper's willingness to show judge-disagreement cases in Appendix A.1. However, the headline conclusion depends on the validity of the GPT-4o judge and on the interpretability of the reported scores. With no human baseline and a judge known to disagree with humans in at least one concrete case, the comparative and mechanistic conclusions (which models are best, which categories are hardest, whether thinking helps) are not yet established. The benchmark is also not released, which prevents independent verification.

major comments (4)
  1. [§5.3, Appendix A.1, Figs. 10–13] Every score in Tables 1–3 flows through the GPT-4o judge, so the judge's validity is load-bearing. The paper reports 99% consistency with human evaluation on 200 cases, but Appendix A.1 documents a concrete disagreement: GPT-4o marks a response True even though the response adds content ('there are clouds') not present in the human-annotated reference, which the human annotator marks False. The scoring rubric in A.5 explicitly tolerates vague answers, accepts a response if at least one of multiple explanations matches, and instructs the judge to disregard intermediate reasoning. Because GPT-4o is itself an LVLM, its leniency may interact with answer style: longer thinking-mode outputs with multiple candidate explanations may be more likely to contain one acceptable phrase. This could change the Table 2 'thinking does not consistently help' conclusion and the Table 1 rankings. The authors
  2. [§6.1, Table 1] No human accuracy baseline on IllusionReasoning is reported. The abstract's claim that LVLMs' reasoning capabilities 'are not as advanced as claimed' and the paper's characterization of the benchmark as 'challenging' require calibration: if human subjects also score at or below the 24–61% range on the same questions, the low model scores may reflect ambiguity in the task rather than a specifically LVLM deficiency. The authors should collect human accuracy on a representative sample of the actual benchmark questions, split by question type and illusion category, and report it alongside the model results. Without such a baseline, the main headline cannot be interpreted.
  3. [Data availability (throughout)] The central contribution is a new benchmark, but no data release link, annotation files, or evaluation code are provided in the manuscript or appendices. No 'data availability' statement appears. As a result, other researchers cannot reproduce the reported numbers, verify the duplication thresholds, or use IllusionReasoning for future evaluation. A benchmark paper of this type should make the data and judging pipeline publicly available (or at least provide a clear release plan).
  4. [§7.3, Table 3, Appendix A.4] The multi-choice task asks models to 'Select NO MORE THAN TWO best reasons,' and the evaluation criterion is described only as checking whether the annotated illusion categories 'appeared in the output.' This is ambiguous and likely overly lenient: a response that selects two categories with only one correct, or that emits more than two options, may receive credit. Since Table 3 is used to conclude that thinking mode hurts multi-choice reasoning (e.g., Qwen3.5: 69.00 → 66.83), the exact matching criterion (exact set, subset, partial credit, F1) must be stated, and the conclusion should be re-checked under a stricter criterion that does not reward extra guesses.
minor comments (6)
  1. [Table 2] The two rows per model are distinguished only by gray shading, which may not survive monochrome printing. Please add an explicit mode label (e.g., 'thinking' / 'non-thinking') in a dedicated column.
  2. [§7.2, Figure 5] The method for assigning responses to Human / Neutrality / Physical World is not described. Was this done by human annotators, by a model, or by keyword matching? This needs to be specified, because the 'safe response' conclusion depends on the reliability of these labels.
  3. [§4.5, Figure 2] The text says QA pairs are distributed at an approximate 2:2:1 ratio, but Figure 2 should be checked to ensure the counts for Detection / Description / Reasoning are consistent with the stated 'over 3,000 QA pairs.'
  4. [Table 3] The 'Choice' and 'Free-form' columns appear to come from different question subsets; the sample sizes and the exact instruction for the multi-choice task are not given. Please state how many questions each number is based on.
  5. [Appendix A.1] The paper says 'there are few disagreements,' but 99% consistency on 200 samples implies at least one, possibly two, disagreements in the validation set. A more precise statement of the validation sample and the number of disagreements would help.
  6. [Throughout] Typos and spacing issues: 'constructedIllusionReasoning' in the Abstract and Section 1 should be 'constructed IllusionReasoning.' Please also check 'Qwen3.5-235BA22B' etc. in Figure 5.

Circularity Check

0 steps flagged

No circularity: the benchmark's ground truth is human-annotated and the central claim is an empirical evaluation result, not an input-equivalent construction.

full rationale

The paper's derivation chain is: collect real-world illusion images, have human annotators produce verified question-answer pairs, have LVLMs answer those questions, and use GPT-4o as an automated judge with a reported 99% human-consistency rate on 200 sampled cases. The central claim that LVLMs' reasoning capabilities are 'not as advanced as claimed' is an empirical generalization from the measured accuracies in Tables 1-3, not a definitional consequence of the benchmark. No equation or construction step equates the benchmark scores to the inputs: the correct answers are human-annotated physical-world truths, and the judged model outputs are external to the benchmark. There is no parameter fitted to the model outputs that is then relabeled as a prediction, and no load-bearing self-citation or imported uniqueness theorem. The Appendix A.1 disagreement between GPT-4o and human annotators is a measurement-validity caveat rather than circularity: the scores are not forced to match the benchmark by construction. Accordingly, the paper is self-contained with respect to circularity concerns, and the honest finding is no significant circularity (score 0).

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

No invented physical or mathematical entities and no fitted parameters in the derivation sense. The only hand-chosen numbers are benchmark design choices: QA category mix (2:2:1), duplication thresholds (3%/10%), and the 200-case judge-validation sample. Each affects the reported aggregate scores but none is fitted to maximize agreement with the target result. The validity of the whole evaluation rests on the four axioms above, especially the reliability of GPT-4o as judge, which the appendix partially undermines.

free parameters (3)
  • QA category ratio = ~2:2:1 detection:description:reasoning
    Hand-chosen mix in §4.5; aggregate accuracy and the 'reasoning' headline depend on this ratio, with no sensitivity analysis.
  • Novelty/duplication thresholds = <3% vs IllusionVQA, <10% vs IllusionBench+
    Thresholds chosen in §4.5 to claim novelty; the method for computing duplication is not described.
  • Judge-validation sample size = 200 cases
    §5.3 uses 200 sampled cases to claim 99% human consistency; no confidence interval is reported, and Appendix A.1 shows a concrete disagreement.
axioms (4)
  • domain assumption Visual illusions are a valid diagnostic for jointly measuring LVLMs' perception and reasoning.
    Stated in §1 and §3.2; the entire benchmark's rationale depends on this premise.
  • domain assumption Human annotators' initial misperception and physical-world metadata are correct ground truth; retaining only images that align with documented misperception does not bias the benchmark.
    §4.4 filtering step assumes annotator consensus equals the documented misperception and that physical reality is accurately captured by metadata.
  • domain assumption Real-world search-engine images with low duplication rates against prior benchmarks are novel to LVLMs and therefore test perception rather than memorization.
    §4.3 and §4.5; no test is run to show the models have not seen similar images during training.
  • domain assumption GPT-4o is a reliable semantic judge of free-form answers.
    §5.3 invokes 99% consistency on 200 cases; Appendix A.1 shows a case where GPT-4o over-accepts, so this assumption is partially contradicted by the paper's own evidence.

pith-pipeline@v1.3.0-daily-deepseek · 13365 in / 14340 out tokens · 134737 ms · 2026-08-01T02:02:13.049036+00:00 · methodology

0 comments
read the original abstract

Large Vision Language Models have integrated reasoning capabilities, elevating cognitive performance to new levels. However, existing evaluations either focus solely on perception or rely on specific domains such as maths or coding. Evaluation for reasoning capabilities that align with an open-world environment is still required, especially one that considers perception and reasoning jointly. To bridge this gap, we propose to evaluate LVLMs by exploiting visual illusions as a diagnostic tool. Visual illusions are phenomena in which the human visual system misinterprets objective signals, resulting in an understanding that deviates from reality. We constructed IllusionReasoning, a benchmark of illusion images collected from the real world, incorporating diverse annotated question-answer pairs. Based on IllusionReasoning, we show that the reasoning capabilities of a wide range of LVLMs are not as advanced as claimed. Our work provides new insights into LVLMs and offers future direction for optimisation.

Figures

Figures reproduced from arXiv: 2607.27747 by Da Li, Jianing Li, Jiaqing Lyu, Kexin Tang, Liangjie Zhao, Rong Yin, Yulan Hu, Zecheng Fang.

Figure 1
Figure 1. Figure 1: Overview of IllusionReasoning Construction. Images in IllusionReasoning are collected from search engines to ensure authenticity. The annotation consists of two stages: (1) Content Analysis, which involves the determination of illusion types by comparing human descriptions with ground-truth reality, and (2) QA Construction, which encompasses the creation of high-quality detection, description, and reasonin… view at source ↗
Figure 3
Figure 3. Figure 3: Comparison between IllusionReasoning and [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 2
Figure 2. Figure 2: Statistics of IllusionReasoning. The inner ring [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Result comparison between Gemini-3.1- Pro, Qwen3.5-397B, and GPT-5.5. The characteristics within the red box indicate that the gray car does not actually exist. 7 Further Analysis 7.1 Can thinking help LVLMs to recognize illusions? Recent LVLMs have integrated reasoning capabil￾ities through introducing thinking mode, demon￾strating performance improvements on different evaluations. Whether thinking is eff… view at source ↗
Figure 5
Figure 5. Figure 5: Comparison of LVLMs’ preference for align [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: A case where an LLM as a judge differs from [PITH_FULL_IMAGE:figures/full_fig_p011_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: An example of a neutral output. Taking Qwen [PITH_FULL_IMAGE:figures/full_fig_p011_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: An example of reasoning that has negative [PITH_FULL_IMAGE:figures/full_fig_p012_8.png] view at source ↗
Figure 11
Figure 11. Figure 11: Evaluation Prompt for Description Ques￾tions. Reasoning Questions For reasoning questions (e.g., "why" and "how"): 1. If the model’s answer includes multiple explanations or conclusions, it is correct as long as at least one matches the reference answer. 2. Respond with True if the semantic mean￾ing of the model’s final result matches the reference answer; otherwise, respond with False [PITH_FULL_IMAGE:f… view at source ↗
Figure 12
Figure 12. Figure 12: Evaluation Prompt for Reasoning Questions. [PITH_FULL_IMAGE:figures/full_fig_p012_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: General Rules for All Questions [PITH_FULL_IMAGE:figures/full_fig_p013_13.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

14 extracted references · 3 linked inside Pith

  1. [1]

    If the model’s answer lacks a clear judg- ment word, respond with False

  2. [2]

    how many

    If the model’s answer includes a judg- ment word: i. Respond with True if it matches the refer- ence answer. ii. Respond with False if it does not match. Figure 10: Evaluation Prompt for Detection Questions. Description Questions For special questions (e.g., "how many", "what", "which", etc.):

  3. [3]

    Judgement

    Deepseek-r1: Incentivizing reasoning capa- bility in llms via reinforcement learning.Preprint, arXiv:2501.12948. Yuhao Dong, Zuyan Liu, Hai-Long Sun, Jingkang Yang, Winston Hu, Yongming Rao, and Ziwei Liu. 2025. Insight-v: Exploring long-chain visual reasoning with multimodal large language models.Preprint, arXiv:2411.14432. Alexey Dosovitskiy. 2020. An i...

    Pith/arXiv arXiv 2025

  4. [4]

    English), case, punctuation, gram- mar, or word order

    Ignore differences in language (e.g., Chi- nese vs. English), case, punctuation, gram- mar, or word order

  5. [5]

    Figure 13: General Rules for All Questions

    Disregard intermediate reasoning or steps in the model’s answer and evaluate only the final result or conclusion. Figure 13: General Rules for All Questions

  6. [6]

    Respond with False if the model’s answer lacks a clear conclusion

  7. [7]

    why" and

    If the model’s answer includes a clear con- clusion, compare its core information with the reference answer: i. Respond with True if the semantic mean- ing matches. ii. Respond with False if it does not match. Figure 11: Evaluation Prompt for Description Ques- tions. Reasoning Questions For reasoning questions (e.g., "why" and "how"):

  8. [8]

    If the model’s answer includes multiple explanations or conclusions, it is correct as long as at least one matches the reference answer

  9. [9]

    Figure 12: Evaluation Prompt for Reasoning Questions

    Respond with True if the semantic mean- ing of the model’s final result matches the reference answer; otherwise, respond with False. Figure 12: Evaluation Prompt for Reasoning Questions. General Rules for All Questions

  10. [10]

    The model’s answer must not contradict the reference answer

  11. [11]

    Vague answers are acceptable if they in- clude key information from the reference answer and do not introduce errors or con- tradictions

  12. [12]

    Focus on whether the semantic meaning of the model’s answer matches the reference answer

  13. [2024]

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and Feng Zhao

    Hallucination of multimodal large language models: A survey.arXiv preprint arXiv:2404.18930. Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and Feng Zhao. 2024a. Are we on the right way for evaluating large vision-language models?Preprint, arXiv:2403.20330. Zhe Chen, Jiannan Wu, Wenhai ...

    Pith/arXiv arXiv

  14. [2025]

    OpenCompass Contributors

    Simplevqa: Multimodal factuality evaluation for multimodal large language models.Preprint, arXiv:2502.13059. OpenCompass Contributors. 2023. Opencompass: A universal evaluation platform for foundation models. https://github.com/open-compass/ opencompass. DeepSeek-AI. 2026. Deepseek-v4 technical re- port. https://github.com/deepseek-ai/ DeepSeek-V4. Access...

    Pith/arXiv arXiv 2023