REVIEW 3 major objections 3 minor 39 references
MLLM illusion failures come from high-frequency attention bias, and a plug-and-play multi-scale filtering strategy restores accuracy from 13% to 84%.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-02 17:32 UTC pith:LS2XCLJ4
load-bearing objection Useful new illusion benchmark and a plausible training-free fix, but the headline accuracy jump is measured against a weaker prompt than the one SMSP gets. the 3 major comments →
SMSP: A Plug-and-Play Strategy of Multi-Scale Perception for MLLMs to Perceive Visual Illusions
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
On the paper's own terms, the central discovery is the high-frequency attention bias: compared with clean images, illusion images have clearly more spectral energy in middle- and high-frequency bands, and the CLIP-style visual encoder shifts its attention from character regions to background regions on illusion images, with high-attention coverage of character areas dropping below 60%. The paper argues that hidden characters are encoded in relatively lower frequencies, so the model is drawn to the wrong signal. SMSP operationalizes this by simulating human squinting (FFT-based low-pass filtering with cutoff l) and viewing from afar (downscaling by factor s and padding to a white canvas), gen
What carries the argument
The load-bearing machinery is the Perception Module with two operations: (1) high-frequency filtering, an FFT-based low-pass filter that zeroes all frequency components above threshold l*min(H,W), simulating squinting; and (2) spatial rescaling, downscaling the image by factor s and pasting it centered on a white canvas, simulating viewing from afar. The Multi-Scale Strategy then produces K such processed variants with parameters chosen as a geometric progression between empirically anchored boundaries (l1=0.012, s1=0.1 for large characters; lK=0.05, sK=0.4 for small ones), and the original image is added as an extra input to preserve standard capabilities. The variants are jointly fed to th
Load-bearing premise
The load-bearing assumption is that the hidden characters' identifying information lives mostly in lower-frequency bands than the distracting background, so zeroing all frequencies above l=0.012*min(H,W) removes the distraction while leaving the character readable; the paper compares whole-image spectral energy but does not measure the frequency content of the character regions directly.
What would settle it
Measure the spectral energy within the character regions of IlluChar images (not just the whole image) and verify that a substantial fraction of the character's discriminative energy lies below the cutoff; alternatively, apply SMSP's filtering stage alone at l=0.012 to the original clean character images and check whether humans or OCR can still read them—if the filtered characters become unrecognizable, the mechanism cannot be frequency-separation as claimed.
If this is right
- If the central claim is correct, MLLM vulnerability to hidden-pattern illusions is a perception-stage defect, not a knowledge or capacity deficit; inference-time preprocessing can largely close the human-model gap.
- SMSP improves accuracy across all six evaluated MLLMs—open- and closed-source—across noise and semantic backgrounds and across large, medium, and small character scales.
- The approach generalizes beyond characters: accuracy rises on illusion images embedding animals, FashionMNIST, MNIST, and harmful patterns (for example, 97.5% versus 34.0% vanilla on animal illusions).
- Standard VQA performance (SimpleVQA, MMStar, RealWorldQA) is preserved when the original image is included; removing the original-input branch causes a sharp drop, showing that branch is essential.
- Fine-tuning on IlluChar yields limited in-distribution gains (11.6% to 22.9% for Qwen3-VL-4B) and does not generalize to other patterns, whereas SMSP improves both in-distribution and cross-pattern accuracy.
Where Pith is reading between the lines
- The effectiveness of a single aggressive cutoff (l=0.012) for large characters suggests the hidden-character signal is band-limited; a direct test would measure spectral energy inside character regions specifically, which the paper does not report.
- High-frequency attention bias may be a general property of ViT-based encoders, so similar frequency-aware preprocessing could aid MLLMs on other fine-grained perception tasks—camouflage, degraded documents, low-texture scenes—and inexpensively harden automated moderation against illusion-based attacks.
- The fixed geometric parameter schedule raises an open question: an adaptive variant—choosing l and s per image via a lightweight predictor—might extend gains to unseen backgrounds or character styles with fewer input tokens.
- Because the gains are largest for large characters, the bias is likely frequency-dependent; progressively raising the cutoff for small characters could reveal a trade-off between background suppression and character preservation.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces IlluChar, a dataset of character-based hidden-pattern visual illusions, and claims that MLLMs fail on such images because of a 'high-frequency attention bias': high-frequency background textures distract the visual encoder and draw attention away from the hidden characters. To mitigate this, the authors propose SMSP, a plug-and-play preprocessing strategy that applies low-pass filtering and spatial downscaling to produce multiple perception-adjusted variants of the input image, which are fed to the MLLM together with the original image. Experiments on six MLLMs report large accuracy gains on IlluChar (e.g., Qwen3-VL-8B-Instruct from 13.0% to 84.0%), with additional experiments on other illusion types and standard VQA benchmarks.
Significance. If the reported gains are real and attributable to the proposed mechanism, the paper makes a valuable contribution: it provides a new challenging benchmark, a concrete mechanistic hypothesis, and a training-free, plug-and-play intervention that could be applied broadly to MLLMs. The paper includes useful supporting analyses (attention-map variance, spectral energy comparisons, ablations, cost measurements, generalization tests) and releases code. However, the headline comparison is currently confounded by a prompt mismatch between SMSP and the baselines, and the spectral-separation evidence is indirect. These issues must be resolved before the central claims can be accepted.
major comments (3)
- [Appendix D; §6.1–6.2] The main comparison in Table 2 is confounded by prompt format. Vanilla, Filtered, and Blur with Histogram use the default single-image prompt 'There is a [hidden_type] in the image, what is it?', while SMSP receives 'I provide four views of the SAME image... with the help of the views, what is it?' The SMSP prompt tells the model that four images are related, that the hidden content is the same, and that the views should be used jointly. Thus the 13.0%→84.0% gain on Qwen3-VL-8B (and the analogous gaps in Tables 5 and 9) may be caused by the more informative prompt rather than by the proposed perceptual processing. The paper needs a matched-prompt control, e.g., the SMSP prompt with the original image repeated four times, and ideally SMSP-processed images evaluated under the default prompt. Without this control, the central claim is not established.
- [§4 RQ1; Figure 3; Algorithm 1] The mechanistic claim that hidden characters live in lower frequency bands is inferred from whole-image spectral energy of original versus illusion images. This does not measure the frequency content of character regions versus background regions. If character strokes carry substantial high-frequency energy, the λ=0.012 cutoff in the strongest variant would remove the signal the model needs. Please provide a local spectral analysis using the known character masks (IlluChar construction provides them), or a character/background region comparison. Additionally, the pseudocode in Algorithm 1 applies the low-pass cutoff to sqrt(u²+v²) after FFTShift, which is not a centered radial low-pass filter; as written, it would zero out the center of the shifted spectrum. If the implementation differs, the pseudocode should be corrected.
- [§6.3; Figures 6–7; Table 2] The free parameters (λ1, s1, λK, sK, K) are selected by grid search on validation subsets from the same IlluChar benchmark family and then evaluated on IlluChar; no independent held-out split or nested validation is reported. This creates a risk that part of the large improvement reflects tuning to the test distribution. The authors should report results with fixed default parameters or use a leave-one-background-out / leave-one-scale-out validation. In addition, all accuracies are point estimates; with per-cell sample sizes (e.g., Table 7), some differences may not be significant. Report confidence intervals or significance tests for the headline comparisons.
minor comments (3)
- [Table 3] The header 'Origin Illusion Large Medium Small' is ambiguous: the SMSP row has only four numeric entries for five labeled columns, and the ablation rows appear to mix overall illusion accuracy with scale-wise values. Add explicit column names (e.g., Origin | Illusion-Avg | Large | Medium | Small) and ensure every row has the same number of entries.
- [Figure 3] The caption does not define the frequency bands or state whether the plotted curves are averages over classes. Adding axis labels, error bars, and a legend would make the spectral comparison interpretable.
- [§2; Appendix A] The pilot study is placed in Appendix A but is cited as 'detailed in Appendix A' in Section 1; consider referencing the specific figure. Also, the related work section would benefit from a sentence distinguishing IlluChar's construction from IllusoryVQA's AI-generated scenes, since the latter is also used at inference.
Circularity Check
No significant circularity: the mechanism analysis, method design, and benchmark evaluation are not equivalent to the paper's own inputs.
full rationale
The paper's derivation chain is not circular. The 'high-frequency attention bias' mechanism is inferred from two independent measurements—spectral energy analysis (Eq. 1–2, Figure 3) and attention-map variance (Figure 4)—before the mitigation method is introduced. The Perception Module operationalizes this mechanism through low-pass filtering and spatial rescaling (Algorithm 1), and the Multi-Scale Strategy is justified by scale-wise ablations (Table 3), not by construction. The headline accuracy gains are measured on IlluChar, a dataset constructed for this study, but the boundary parameters (λ1,s1) and (λK,sK) are chosen on a validation subset via grid search (Section 6.3) and then applied to the test evaluation; this is standard hyperparameter selection, not a fitted value being renamed as a prediction. The paper's self-citations are absent; references to prior work such as [1], [25], and [28] are external. The main experimental-design concern—that Appendix D uses a more informative prompt for SMSP than for the Vanilla/Filtered/Blur baselines—is a potential confound that could affect attribution of the gain to image preprocessing, but it is not a case where a prediction reduces to its inputs by definition or by construction. Similarly, the acknowledged limitation in Appendix I (fixed parameters, no per-image adaptation) is a scope restriction, not a circular dependency. Under the specified rules, no circular step can be exhibited, so the circularity score is 0.
Axiom & Free-Parameter Ledger
free parameters (5)
- λ1 (low-pass cutoff, strongest variant) =
0.012
- s1 (downscale factor, strongest variant) =
0.1
- λK (low-pass cutoff, weakest variant) =
0.05
- sK (downscale factor, weakest variant) =
0.4
- K (number of SMSP variants) =
3
axioms (5)
- domain assumption Hidden characters are primarily encoded in relatively lower-frequency components than the distracting backgrounds.
- domain assumption CLIP-ViT attention (via Chefer et al. explainability) is representative of how all six MLLMs allocate visual attention.
- domain assumption The human perceptual metaphors 'squinting' and 'viewing from afar' map to low-pass filtering and downscaling-padding.
- domain assumption The hybrid string-matching + GPT-4o evaluator is a valid proxy for human judgment.
- domain assumption Parameters tuned on the validation subsets generalize to the IlluChar test images.
read the original abstract
Recent works have shown that multimodal large language models (MLLMs) are highly vulnerable to hidden-pattern visual illusions, where the hidden content is imperceptible to models but obvious to humans. This deficiency highlights a perceptual misalignment between current MLLMs and humans, and also introduces potential safety concerns. To systematically investigate this failure, we introduce IlluChar, a comprehensive and challenging illusion dataset, and uncover a key underlying mechanism for the models' failure: high-frequency attention bias, where the models are easily distracted by high-frequency background textures in illusion images, causing them to overlook hidden patterns. To address the issue, we propose the Strategy of Multi-Scale Perception (SMSP), a plug-and-play framework that aligns with human visual perceptual strategies. By suppressing distracting high-frequency background signals, SMSP generates images closer to human perception. Our experiments demonstrate that SMSP significantly improves the performance of all evaluated MLLMs on illusion images, for instance, increasing the accuracy of Qwen3-VL-8B-Instruct from 13.0% to 84.0%. Our work provides novel insights into MLLMs' visual perception, and offers a practical and robust solution to enhance it. Our code is publicly available at https://github.com/Tujz2023/SMSP.
Figures
Reference graph
Works this paper leans on
-
[1]
Edward H Adelson, Charles H Anderson, James R Bergen, Peter J Burt, and Joan M Ogden. 1984. Pyramid methods in image processing.RCA engineer29, 6 (1984), 33–41
1984
-
[2]
Anthropic. 2025. Introducing Claude Sonnet 4.5. https://www.anthropic.com/ news/claude-sonnet-4-5
2025
-
[3]
Tejas Anvekar, Fenil Bardoliya, Pavan K Turaga, Chitta Baral, and Vivek Gupta
-
[4]
Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan ...
Pith/arXiv arXiv 2025
-
[5]
Ryan Burgert, Xiang Li, Abe Leite, Kanchana Ranasinghe, and Michael Ryoo
-
[6]
Hila Chefer, Shir Gur, and Lior Wolf. 2021. Generic attention-model explainability for interpreting bi-modal and encoder-decoder transformers. InProceedings of the IEEE/CVF international conference on computer vision. 397–406
2021
-
[7]
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al . 2024. Are we on the right way for evaluating large vision-language models?Advances in Neural Information Processing Systems37 (2024), 27056–27087
2024
-
[8]
Xianfu Cheng, Wei Zhang, Shiwei Zhang, Jian Yang, Xiangyuan Guan, Xianjie Wu, Xiang Li, Ge Zhang, Jiaheng Liu, Yuying Mai, et al. 2025. Simplevqa: Multimodal factuality evaluation for multimodal large language models. InProceedings of the IEEE/CVF International Conference on Computer Vision. 4637–4646
2025
-
[9]
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multi- modality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261(2025)
Pith/arXiv arXiv 2025
-
[10]
Ziqi Ding, Gelei Deng, Yi Liu, Junchen Ding, Jieshan Chen, Yulei Sui, and Yuekang Li. 2025. IllusionCAPTCHA: A CAPTCHA based on visual illusion. InProceedings of the ACM on Web Conference 2025. 3683–3691
2025
-
[11]
Jinyu Fan and Yi Zeng. 2023. Challenging deep learning models with image distortion based on the abutting grating illusion.Patterns4, 3 (2023)
2023
-
[12]
Daniel Geng, Inbum Park, and Andrew Owens. 2024. Visual anagrams: Generating multi-view optical illusions with diffusion models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 24154–24163
2024
-
[13]
Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. 2024. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognitio...
2024
-
[14]
Elad Hirsch and Ayellet Tal. 2020. Color visual illusions: A statistics-based computational model.Advances in neural information processing systems33 (2020), 9447–9458
2020
-
[15]
Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi, Junhui Ji, Lihang Pan, et al. 2025. Glm-4.5 v and glm-4.1 v-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning.arXiv preprint arXiv:2507.01006(2025)
Pith/arXiv arXiv 2025
-
[16]
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Liang Wang, Weizhu Chen, et al. 2022. Lora: Low-rank adaptation of large language models.Iclr1, 2 (2022), 3
2022
-
[17]
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 2002. Gradient- based learning applied to document recognition.Proc. IEEE86, 11 (2002), 2278– 2324
2002
-
[18]
Shanchuan Lin, Anran Wang, and Xiao Yang. 2024. Sdxl-lightning: Progressive adversarial diffusion distillation.arXiv preprint arXiv:2402.13929(2024)
Pith/arXiv arXiv 2024
-
[19]
Shuai Liu, Weilin Pu, Chongling Xu, Zishuo Huang, Qian Li, Hang Wang, Chen- hao Lin, and Chao Shen. 2024. A Comprehensive Survey of Multimodal Large Language Models: Concept, Application and Safety. (2024)
2024
-
[20]
Dominique Makowski, Zen J Lau, Tam Pham, W Paul Boyce, and SH Annabel Chen. 2021. A parametric framework to generate visual illusions using python.Perception50, 11 (2021), 950–965
2021
-
[21]
Artemis Panagopoulou, Coby Melkin, and Chris Callison-Burch. 2024. Evaluating vision-language models on bistable images. InProceedings of the Workshop on Cognitive Modeling and Computational Linguistics. 8–29
2024
-
[22]
Yiting Qu, Ziqing Yang, Yihan Ma, Michael Backes, Savvas Zannettou, and Yang Zhang. 2025. Hate in Plain Sight: On the Risks of Moderating AI-Generated Hateful Illusions. InProceedings of the IEEE/CVF International Conference on Computer Vision. 19617–19627
2025
-
[23]
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. InInternational conference on machine learning. PmLR, 8748–8763
2021
-
[24]
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 10684–10695
2022
-
[25]
Mohammadmostafa Rostamkhani, Baktash Ansari, Hoorieh Sabzevari, Farzan Rahmani, and Sauleh Eetemadi. 2025. Illusory VQA: Benchmarking and enhancing multimodal models on visual Illusions. InProceedings of the Computer Vision and Pattern Recognition Conference. 2995–3004
2025
-
[26]
Haz Sameen Shahgir, Khondker Salman Sayeed, Abhik Bhattacharjee, Wasi Uddin Ahmad, Yue Dong, and Rifat Shahriyar. 2024. Illusionvqa: A challenging optical illusion dataset for vision language models.arXiv preprint arXiv:2403.15952 (2024)
Pith/arXiv arXiv 2024
-
[27]
Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, et al
-
[28]
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems35 (2022), 24824–24837. Preprint, 2026, Tu et al
2022
-
[29]
xAI. 2024. RealworldQA: A Benchmark for Real-World Spatial Understanding. https://huggingface.co/datasets/xai-org/RealworldQA Accessed: 2025-04-26
2024
-
[30]
Openai gpt-5 system card.arXiv preprint arXiv:2601.03267(2025)
Pith/arXiv arXiv 2025
-
[31]
Xueyi Ye, Mingcong Sui, Maosheng Zeng, Zhuo Han, and Hao Wang. 2024. Frequency domain characteristics and optimization of image generation for GANs. (2024)
2024
-
[32]
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. 2023. Adding conditional con- trol to text-to-image diffusion models. InProceedings of the IEEE/CVF international conference on computer vision. 3836–3847
2023
-
[33]
Han Xiao, Kashif Rasul, and Roland Vollgraf. 2017. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms.arXiv preprint arXiv:1708.07747(2017)
Pith/arXiv arXiv 2017
-
[34]
Yiming Zhang, Zicheng Zhang, Xinyi Wei, Xiaohong Liu, Guangtao Zhai, and Xiongkuo Min. 2025. IllusionBench: A Large-scale and Comprehensive Bench- mark for Visual Illusion Understanding in Vision-Language Models.arXiv preprint arXiv:2501.00848(2025). SMSP: A Plug-and-Play Strategy of Multi-Scale Perception for MLLMs to Perceive Visual Illusions Preprint, ...
Pith/arXiv arXiv 2025
-
[36]
Yichi Zhang, Jiayi Pan, Yuchen Zhou, Rui Pan, and Joyce Chai. 2023. Grounding visual illusions in language: Do vision-language models perceive illusions like humans?arXiv preprint arXiv:2311.00047(2023)
Pith/arXiv arXiv 2023
-
[38]
Ignore the sharp, high frequency details, textures and noise in the image
Imagine squinting your eyes or slightly blurring your vision. Ignore the sharp, high frequency details, textures and noise in the image
-
[39]
You can resize the image smaller in your mind to get a global view of the image
Imagine viewing the image from a long distance. You can resize the image smaller in your mind to get a global view of the image. You can combine the two strategies to enhance your perception of t- he hidden character. Now, please analyze the image carefully, and identify the hidden [hidden_type]. For SMSP, each sample is processed into four input images f...
2026
-
[2024]
InACM SIGGRAPH 2024 Conference Papers
Diffusion illusions: Hiding images in plain sight. InACM SIGGRAPH 2024 Conference Papers. 1–11
2024
-
[2025]
The Perceptual Observatory Characterizing Robustness and Grounding in MLLMs.arXiv preprint arXiv:2512.15949(2025)
arXiv 2025
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.