REVIEW 4 major objections 4 minor 2 cited by
IllusionCAPTCHA: A CAPTCHA based on Visual Illusion
T0 review · 4 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read IllusionCAPTCHA defeats LLMs 100% of the time while 86.95% of humans pass on the first attempt.
desk verdict A genuinely new CAPTCHA mechanism with a real proof-of-concept, but the headline '100% deception' claim is conditioned on a non-adversarial fixed prompt and a curated test set, so the paper overstates its security. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the illusion diffusion model (built on ControlNet) that fuses a base image with a user prompt, so the output image displays the prompt's content on its surface while preserving the base content underneath; the most confusing image is selected by lowest cosine similarity to the base. The second mechanism is the four-option multiple-choice frame with a deliberately verbose decoy describing the prompt's surface content, plus an inducement prompt ('Tell us the true and detailed answer of this image') that triggers LLM hallucination and length bias. Together they make the CAPTCHA a forced error for the tested LLMs and a quick recognition task for humans.
What would settle it
Run the same 30 illusionary CAPTCHA images against GPT-4o or Gemini with a prompt such as 'Ignore the surface decoration; describe any hidden or embedded content, and then choose the option that matches the hidden content,' plus a simple crop or contrast adjustment, and measure the success rate; any rate above 0% would falsify the 100% deception claim.
Extended reading notes
Core claim
The authors claim to have built the first illusion-based CAPTCHA that instantiates the 'Human-Easy but AI-Hard' paradigm: an image is blended with a decoy prompt so that the generated picture looks like the prompt to an AI vision system but still reveals the original content to a human. Four multiple-choice options are offered, and a step-by-step 'inducement prompt' instructs the model to give the true detailed answer, which exploits the models' tendency to produce long detailed descriptions and steers them to the wrong option. In evaluation, neither GPT-4o nor Gemini 1.5 Pro 2.0 solved any of the 30 illusionary CAPTCHAs under zero-shot or chain-of-thought prompting, and the inducement prompt made both models pick the intended wrong option in 100% of cases; 86.95% of human participants passed on the first attempt.
Load-bearing premise
The 100% deception claim assumes the attacker uses the exact inducement prompt and the fixed four-option layout with no prompt rewriting and no image preprocessing; if an attacker simply asks the model to ignore the decorative text and identify hidden content, the reported rate has no support.
Editorial extensions
If this is right
- If the result holds, existing LLM-based CAPTCHA solvers cannot pass IllusionCAPTCHA under the evaluated interaction, restoring a usable security layer for web services.
- The inducement-prompt design becomes a template: any multimodal multiple-choice challenge can be structured to steer LLMs toward a predictable wrong answer.
- Because 86.95% of humans pass on the first attempt, IllusionCAPTCHA would improve usability relative to reasoning-based CAPTCHAs, which the paper finds require a second attempt for most users.
- The empirical finding that LLMs and humans make the same mistakes on existing CAPTCHAs implies that separate designs, not harder vision tasks, are needed to tell them apart.
Reading between the lines
- The 100% deception rate is an upper bound under the exact inducement prompt and four-option layout; an attacker who rewrites the prompt to ignore the surface description, or preprocesses the image, may defeat it, and that scenario is not tested.
- The underlying mechanism suggests a benchmark: illusionary images could be used to measure whether a vision model truly perceives semantics or only matches surface text, a testable extension beyond CAPTCHAs.
- An arms race is likely: fine-tuning on illusionary images or adding a second model to check options could close the gap, so the security claim may be time-limited.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper reports two contributions. First, it presents an empirical study of two multimodal LLMs (GPT-4o and Gemini 1.5 Pro 2.0) on text-based, image-based, and reasoning-based CAPTCHAs under zero-shot and chain-of-thought prompting, together with a user study of the number of attempts humans need. The findings are that LLMs solve most text-based CAPTCHAs, do poorly on reasoning-based CAPTCHAs, and improve with CoT, while human users often need more than one attempt, especially on reasoning CAPTCHAs. Second, it introduces IllusionCAPTCHA, in which an illusion diffusion model blends a base image with a user prompt, four multiple-choice options are generated (one correct, one matching the generation prompt, and two verbose distractors), and an 'inducement prompt' is used to lure LLMs into selecting the distractors. The evaluation reports that both tested LLMs fail all 30 IllusionCAPTCHA images (100% deception in RQ4/RQ5), while 86.95% of 23 human participants pass on the first attempt (RQ6). The abstract concludes that IllusionCAPTCHA 'can effectively deceive LLMs 100% of the time' and outperforms other CAPTCHA systems in usability.
Significance. If the claims held broadly, the paper would make a useful contribution to the CAPTCHA arms race: visual illusions are a plausible source of human-AI asymmetry, the idea of deliberately engineered misleading options is creative, and the initial empirical study of LLMs across CAPTCHA categories is relevant to the community. The paper also provides a first attempt at measuring human first-attempt success rates across CAPTCHA types, which is a useful data point. However, the evaluation of IllusionCAPTCHA is narrow: only two models, one fixed prompt, one fixed option layout, and a test set selected for maximal deviation from the base image. The headline 100% figure is therefore an artifact of the particular evaluation setup rather than a demonstrated property of the proposed mechanism. The human samples are also small (10 for RQ3, 23 for RQ6), and the usability comparison to other CAPTCHA systems is not backed by a controlled comparison in this paper.
major comments (4)
- [Abstract and §6.3, Table 5] The central claim that IllusionCAPTCHA 'can effectively deceive LLMs 100% of the time' is conditional on the exact inducement prompt from §5.3 and the fixed four-option layout from §5.2. §3's threat model only excludes reverse engineering, JavaScript decompiling, and direct code analysis; it does not forbid an attacker from rewriting the prompt, cropping or inpainting the image, or using a second model to check the options. No such adaptive attacks are tested. An attacker who simply asks the model to ignore the decorative description and identify the hidden content is outside the tested evaluation, so the 100% figure does not support a general AI-hard claim.
- [§5.1 and §6.2, Table 4] The 30 IllusionCAPTCHA test images are selected by generating 50 candidates and keeping the one with the lowest cosine similarity to the base image. This is a form of selection bias: the evaluation does not sample from the distribution of images the generator produces but from the extreme tail that is least similar to the base image. The reported 0% LLM success rate may therefore overstate the deception rate of a randomly generated instance. In addition, the illusion strength is a fixed free parameter (1.5) with no sensitivity analysis, so it is unclear how robust the result is to parameter changes.
- [§5.2, §5.3, and §6.3, Table 5] The option design deliberately includes the exact prompt used to generate the illusion as a distractor, and another distractor is a verbose description of the illusionary element. The paper attributes the LLMs' 100% selection of the predicted option to visual illusions, but the evaluation does not ablate this option design. Without testing options with short, generic distractors or without the prompt-matching distractor, it remains possible that the LLMs are mostly responding to the length- and prompt-matching bias of the options rather than to the illusionary content. The contribution of the 'structured step-by-step method' is therefore not isolated from the confounding of option content.
- [§6.1 and §6.4, Tables 3 and 6] The human usability evidence is thin: RQ3 uses 10 participants and RQ6 uses 23, with no statistical testing or confidence intervals. Moreover, the RQ3 task (perceiving and describing an illusion) is not the same as the RQ6 task (selecting one of four multiple-choice options), so the 83%/88% visibility figures in Table 3 do not by themselves validate the 86.95% first-attempt pass rate in Table 6. The abstract's statement that IllusionCAPTCHA is 'outperforming other CAPTCHA systems' is not supported by any comparative user study in the paper, since no other CAPTCHA was presented to the same participants under the same protocol.
minor comments (4)
- [§4.1] There are several typos and infelicities, e.g., 'capthchas' in the dataset collection paragraph, 'Overloaping' in Table 1, 'incoperate' in §5, and 'multimodel' in §5. The paper would benefit from a proofreading pass.
- [§4.1 and §6.2] The datasets and evaluation protocols are described at a high level, but it is not always clear how the multiple-choice options in RQ4 were presented to the LLMs (i.e., whether the model was given the four options or asked to answer openly). Clarifying the exact input format for each RQ would improve reproducibility.
- [§5.1] The paper states that 50 images are generated 'at a fixed illusion strength level of 1.5—an optimal value for human identification in this context,' but no evidence is provided for this optimality. A small sensitivity analysis or a reference to prior work would be useful.
- [§7] The discussion of adversarial attacks and cross-cultural adaptability is qualitative and does not include any experiments. The claim that adversarial images lack transferability is only supported by a reference, not by experiments on the proposed system.
Circularity Check
No significant circularity; the paper's claims rest on empirical measurements rather than on derivations that reduce to their own inputs.
full rationale
The paper's quantitative claims are empirical measurements rather than derived results that collapse into their inputs. Section 4 reports measured LLM and human performance on existing CAPTCHAs; Section 5 motivates a design from those observations, including an inducement prompt and a deliberately verbose distractor, citing an external source ([15]) for LLM preference for long descriptions. Sections 6.2 and 6.3 then measure whether that design achieves its goals, and the reported 0% identification rate and 100% inducement rate are actual LLM responses, not consequences forced by construction. The image-selection heuristic in Section 5.1 (choosing the lowest cosine similarity to the base image as a difficulty proxy) does not by itself determine the later success rates; those rates come from running the models on the selected images. The self-citations ([7], [8]) are contextual and not load-bearing; no uniqueness theorem or author-imported ansatz carries the argument. The conditional nature of the 100% claim, which depends on a fixed four-option layout and a fixed inducement prompt, is a limitation on external validity rather than a circularity in the paper's derivation chain. Overall, no significant circularity is present.
Assumptions & free parameters
free parameters (1)
- illusion_strength =
1.5
assumptions (5)
- domain assumption Current multimodal LLMs systematically fail to identify hidden content in illusionary images.
- domain assumption IllusionDiffusion blending preserves enough base-image content for humans to recognize while hiding it from AI.
- domain assumption LLMs tend to select long, detailed descriptions and can be lured by inducement prompts into predictable wrong answers.
- domain assumption Human users know the named base content, such as characters or landmarks.
- ad hoc to paper An attacker will use the unmodified inducement prompt and the fixed four-option interface.
Cite this review
Pith. "Pith review of IllusionCAPTCHA: A CAPTCHA based on Visual Illusion." pith.science (2026). https://pith.science/paper/DSOV5EGG
@misc{pith2026250205461,
author = {Pith},
title = {Pith review of: IllusionCAPTCHA: A CAPTCHA based on Visual Illusion},
year = {2026},
howpublished = {\url{https://pith.science/paper/DSOV5EGG}},
note = {Machine review of arXiv:2502.05461}
}
read the original abstract
CAPTCHAs have long been essential tools for protecting applications from automated bots. Initially designed as simple questions to distinguish humans from bots, they have become increasingly complex to keep pace with the proliferation of CAPTCHA-cracking techniques employed by malicious actors. However, with the advent of advanced large language models (LLMs), the effectiveness of existing CAPTCHAs is now being undermined. To address this issue, we have conducted an empirical study to evaluate the performance of multimodal LLMs in solving CAPTCHAs and to assess how many attempts human users typically need to pass them. Our findings reveal that while LLMs can solve most CAPTCHAs, they struggle with those requiring complex reasoning type of CAPTCHA that also presents significant challenges for human users. Interestingly, our user study shows that the majority of human participants require a second attempt to pass these reasoning CAPTCHAs, a finding not reported in previous research. Based on empirical findings, we present IllusionCAPTCHA, a novel security mechanism employing the "Human-Easy but AI-Hard" paradigm. This new CAPTCHA employs visual illusions to create tasks that are intuitive for humans but highly confusing for AI models. Furthermore, we developed a structured, step-by-step method that generates misleading options, which particularly guide LLMs towards making incorrect choices and reduce their chances of successfully solving CAPTCHAs. Our evaluation shows that IllusionCAPTCHA can effectively deceive LLMs 100% of the time. Moreover, our structured design significantly increases the likelihood of AI errors when attempting to solve these challenges. Results from our user study indicate that 86.95% of participants successfully passed the CAPTCHA on their first attempt, outperforming other CAPTCHA systems.
Figures
Forward citations
Cited by 2 Pith papers
-
Hate in Plain Sight: On the Risks of Moderating AI-Generated Hateful Illusions
Hateful optical illusions generated with Stable Diffusion and ControlNet evade current moderation classifiers (best accuracy 0.245) and vision-language models (best accuracy 0.102), with simple image transformations s...
-
Broken Gates: Re-evaluating Web Bot Defenses in the Age of LLM Agents
Commercial solvers defeat puzzle captchas at near-perfect rates for pennies per thousand, while reCAPTCHA v3's resistance comes from browser-environment authenticity, not human-like behavior.
Reference graph
Works this paper leans on
-
[7]
Gelei Deng, Haoran Ou, Yi Liu, Jie Zhang, Tianwei Zhang, a nd Yang Liu
-
[1]
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad , Ilge Akkaya, Flo- rencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sa m Altman, Shya- mal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)
arXiv 2023
-
[2]
AP123. 2024. https://huggingface.co/spaces/AP123/I llusionDiffusion. https://huggingface.co/spaces/AP123/IllusionDiffusion
work page 2024
-
[3]
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiao dong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609 (2023)
arXiv 2023
-
[4]
Tuhin Chakrabarty, Philippe Laban, Divyansh Agarwal, S maranda Muresan, and Chien-Sheng Wu. 2024. Art or artifice? large language models and the false promise of creativity. In Proceedings of the CHI Conference on Human Factors in Computing Systems. 1–34
work page 2024
-
[5]
ChatGPT. 2024. https://openai.com/index/hello-gpt- 4o/. https://openai.com/index/hello-gpt-4o/
work page 2024
-
[6]
Aolin Che, Yalin Liu, Hong Xiao, Hao Wang, Ke Zhang, and Hong-Ning Dai. 2021. Augmented Data Selector to Initiate Text-Based CAPTCHA Attack. Security and Communication Networks 2021, 1 (2021), 9930608
work page 2021
-
[8]
Ziqi Ding. 2024. IllusionCaptcha. https://sites.goog le.com/view/illusionarycaptcha
work page 2024
Show all 33 references
-
[9]
Valerie Fanelle, Sepideh Karimi, Aditi Shah, Bharath Su bramanian, and Sauvik Das. 2020. Blind and human: Exploring more usable audio {CAPTCHA} designs. In Sixteenth Symposium on Usable Privacy and Security (SOUPS 202 0). 111–125
2020
-
[10]
Yipeng Gao, Haichang Gao, Sainan Luo, Yang Zi, Shudong Z hang, Wenjie Mao, Ping Wang, Yulong Shen, and Jeff Yan. 2021. Research on the sec urity of visual reasoning {CAPTCHA}. In 30th USENIX security symposium (USENIX security 21). 3291–3308
2021
-
[11]
google. 2024. https://www.google.com/recaptcha/ab out/. https://www.google.com/recaptcha/about/
2024
-
[12]
Rich Gossweiler, Maryam Kamvar, and Shumeet Baluja. 20 09. What’s up CAPTCHA? A CAPTCHA based on image orientation. In Proceedings of the 18th international conference on World wide web . 841–850
-
[13]
Rick Gurnsey, G Keith Humphrey, and Paula Kapitan. 1992 . Parallel discrimina- tion of subjective contours defined by offset gratings. Perception & Psychophysics 52 (1992), 263–276
1992
-
[14]
hCaptcha. 2024. https://www.hcaptcha.com/. https:/ /www.hcaptcha.com/
2024
-
[15]
Wenbo Hu, Yifan Xu, Yi Li, Weiyue Li, Zeyuan Chen, and Zhu owen Tu. 2024. Bliva: A simple multimodal llm for better handling of text-r ich visual questions. In Proceedings of the AAAI Conference on Artificial Intelligenc e, Vol. 38. 2256– 2264
2024
-
[16]
Alex Krizhevsky, Geoffrey Hinton, et al. 2009. Learning multiple layers of fea- tures from tiny images. (2009)
2009
-
[17]
Peter Matthews, Andrew Mantel, and Cliff C Zou. 2010. Sce ne tagging: image- based CAPTCHA using image composition and object relations hips. In Proceed- ings of the 5th ACM Symposium on Information, Computer and Co mmunications Security. 345–350
2010
-
[18]
Zahra Noury and Mahdi Rezaei. 2020. Deep-CAPTCHA: a dee p learning based CAPTCHA solver for vulnerability assessment. arXiv preprint arXiv:2006.08296 (2020)
2020 arXiv
-
[19]
Farhad Pourpanah, Moloud Abdar, Yuxuan Luo, Xinlei Zho u, Ran Wang, Chee Peng Lim, Xi-Zhao Wang, and QM Jonathan Wu. 2022. A revie w of gen- eralized zero-shot learning methods. IEEE transactions on pattern analysis and machine intelligence 45, 4 (2022), 4051–4070
2022
-
[20]
Andrew Searles, Yoshimichi Nakatsuka, Ercan Ozturk, A ndrew Paverd, Gene Tsudik, and Ai Enkoji. 2023. An Empirical Study & Evaluation of Modern {CAPTCHAs}. In 32nd usenix security symposium (usenix security 23) . 3081– 3097
2023
-
[21]
Chenghui Shi, Shouling Ji, Qianjun Liu, Changchang Liu , Yuefeng Chen, Yuan He, Zhe Liu, Raheem Beyah, and Ting Wang. 2020. Text captcha i s dead? a large scale deployment and empirical study. In Proceedings of the 2020 ACM SIGSAC conference on computer and communications secu...
2020
-
[22]
Hongda Sun, Weikai Xu, Wei Liu, Jian Luan, Bin Wang, Shuo Shang, Ji-Rong Wen, and Rui Yan. 2024. Determlr: Augmenting llm-based logical reasoning from indeterminacy to determinacy. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volu...
2024
-
[23]
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui W u, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023)
2023 arXiv
-
[24]
Xiwen Teoh, Yun Lin, Ruofan Liu, Zhiyong Huang, and Jin S ong Dong. 2024. {PhishDecloaker}: Detecting {CAPTCHA-cloaked} Phishing Websites via Hy- brid Vision-based Interactive Models. In 33rd USENIX Security Symposium (USENIX Security 24) . 505–522
2024
-
[25]
Luis Von Ahn, Manuel Blum, Nicholas J Hopper, and John La ngford. 2003. CAPTCHA: Using hard AI problems for security. In Advances in Cryptol- ogy—EUROCRYPT 2003: International Conference on the Theory and Applications of Cryptographic Techniques, Warsaw, Poland, May 4–8, 2003 ...
2003
-
[26]
Riidiger von der Heydt and Esther Peterhans. 1989. Mech anisms of contour perception in monkey visual cortex. I. Lines of pattern disc ontinuity. Journal of Neuroscience 9, 5 (1989), 1731–1748
1989
-
[27]
Haipeng Wang, Feng Zheng, Zhuoming Chen, Yi Lu, Jing Gao , and Renjia Wei
-
[28]
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma , Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompti ng elicits reason- ing in large language models. Advances in neural information processing systems 35 (2022), 24824–24837
2022
-
[29]
Zhipeng Wei, Jingjing Chen, Micah Goldblum, Zuxuan Wu, Tom Goldstein, and Yu-Gang Jiang. 2022. Towards transferable adversarial att acks on vision trans- formers. In Proceedings of the AAAI Conference on Artificial Intelligenc e, Vol. 36. 2668–2676
2022
-
[30]
Guixin Ye, Zhanyong Tang, Dingyi Fang, Zhanxing Zhu, Ya nsong Feng, Pengfei Xu, Xiaojiang Chen, and Zheng Wang. 2018. Yet another text ca ptcha solver: A generative adversarial network based approach. In Proceedings of the 2018 ACM SIGSAC conference on computer and communicati...
2018
-
[31]
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. 2023. Addi ng conditional con- trol to text-to-image diffusion models. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision . 3836–3847
2023
-
[2018]
In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
A captcha design based on visual reasoning. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 1967–1971
2018
-
[2024]
arXiv preprint arXiv:2405.07496 (2024)
Oedipus: LLM-enchanced Reasoning CAPTCHA Solver. arXiv preprint arXiv:2405.07496 (2024)
2024 arXiv
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.