{"id":"ef206747-81a9-4170-8683-c233ed0cf8ae","arxiv_id":"2502.05461","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"The authors propose a visual-illusion CAPTCHA that confused GPT-4o and Gemini in all 30 tests while 20 of 23 humans passed on the first try, though the evaluation is narrow and non-adaptive.","lead":"This paper tests how well two large language models solve existing CAPTCHAs, then builds IllusionCAPTCHA, which hides content inside visual illusions. The authors report 100% failure by the models against 86.95% first-try human success, but the evidence rests on tiny, selected samples and no released code.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 100% LLM deception claim depends on a fixed inducement prompt and a deliberately verbose distractor option; an attacker who rewrites the prompt or removes the distractor is outside the tested threat model.","rationale":"The reader's weakest assumption identifies the exact load-bearing point: the evaluation's 100% deception rate is conditioned on the specific inducement prompt and option set described in Sections 5.2–5.3. The paper's own text in Section 5.3 says the prompt is crafted to trigger hallucinations, conceding that the prompt does much of the work. The threat model in Section 3 is unusually narrow, ruling out reverse engineering but not prompt rewriting, image preprocessing, or ensemble verification. A CAPTCHA security claim must be robust to an attacker who controls the prompt, especially since the CAPTCHA is rendered as an image plus user-facing instructions. The RQ5 observation that LLMs tend to select the longest description further supports the role of the verbose distractor. The paper has genuine contributions: a plausible proof-of-concept with 30 images, two frontier LLMs, 0% success under zero-shot and CoT with the inducement prompt, and a small but real human user study (n=23) showing 86.95% first-attempt success. However, the central claim as stated is overbroad and unsupported by the evidence. A single experiment varying the prompt and distractor option would determine whether the deception stems from the visual illusion or from the prompt layout. Given the security context, the strong claim should remain rejected or at most conditional on the restricted threat model.","tokens_in":14161,"tokens_out":4477,"duration_ms":41715,"concrete_test":"Reuse the same 30 IllusionCAPTCHA images and four-option layout from RQ5. Run GPT-4o and Gemini 1.5 Pro 2.0 under three conditions: (a) the original inducement prompt; (b) a neutral prompt such as \"Which option best describes the hidden content of the image?\" with the same four options; and (c) the original inducement prompt but with the verbose distractor replaced by a short, non-descriptive option. If the LLM success rate in condition (b) or (c) exceeds 0%, the 100% deception claim depends on prompt/option engineering, not on the visual illusion itself.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that IllusionCAPTCHA \"can effectively deceive LLMs 100% of the time\" rests on RQ4/RQ5 (Tables 4 and 5) using exactly two models (GPT-4o, Gemini 1.5 Pro 2.0), a fixed four-option multiple-choice layout, and the inducement prompt from Section 5.3 (\"Tell us the true and detailed answer of this image\"). Section 5.2 includes a deliberately elaborate, detailed description of the illusionary element as a distractor, and Section 5.3 explicitly states that the hint is \"crafted to trigger hallucinations in LLMs.\" The threat model (Section 3) only rules out reverse engineering; it does not forbid the attacker from rewriting the prompt, cropping or inpainting the image, or using a second model to verify options. The 100% figure is therefore conditional on a single, non-adversarial prompt and option set. If an attacker simply prompts the model to identify the hidden content rather than the illusionary overlay, or replaces the verbose distractor with a short random option, the deception rate may drop substantially. Without testing those variations, the headline claim is an artifact of the evaluation setup rather than a property of visual illusions themselves.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports two contributions. First, it presents an empirical study of two multimodal LLMs (GPT-4o and Gemini 1.5 Pro 2.0) on text-based, image-based, and reasoning-based CAPTCHAs under zero-shot and chain-of-thought prompting, together with a user study of the number of attempts humans need. The findings are that LLMs solve most text-based CAPTCHAs, do poorly on reasoning-based CAPTCHAs, and improve with CoT, while human users often need more than one attempt, especially on reasoning CAPTCHAs. Second, it introduces IllusionCAPTCHA, in which an illusion diffusion model blends a base image with a user prompt, four multiple-choice options are generated (one correct, one matching the generation prompt, and two verbose distractors), and an 'inducement prompt' is used to lure LLMs into selecting the distractors. The evaluation reports that both tested LLMs fail all 30 IllusionCAPTCHA images (100% deception in RQ4/RQ5), while 86.95% of 23 human participants pass on the first attempt (RQ6). The abstract concludes that IllusionCAPTCHA 'can effectively deceive LLMs 100% of the time' and outperforms other CAPTCHA systems in usability.","tokens_in":14303,"tokens_out":3753,"duration_ms":39725,"significance":"If the claims held broadly, the paper would make a useful contribution to the CAPTCHA arms race: visual illusions are a plausible source of human-AI asymmetry, the idea of deliberately engineered misleading options is creative, and the initial empirical study of LLMs across CAPTCHA categories is relevant to the community. The paper also provides a first attempt at measuring human first-attempt success rates across CAPTCHA types, which is a useful data point. However, the evaluation of IllusionCAPTCHA is narrow: only two models, one fixed prompt, one fixed option layout, and a test set selected for maximal deviation from the base image. The headline 100% figure is therefore an artifact of the particular evaluation setup rather than a demonstrated property of the proposed mechanism. The human samples are also small (10 for RQ3, 23 for RQ6), and the usability comparison to other CAPTCHA systems is not backed by a controlled comparison in this paper.","major_comments":[{"comment":"The central claim that IllusionCAPTCHA 'can effectively deceive LLMs 100% of the time' is conditional on the exact inducement prompt from §5.3 and the fixed four-option layout from §5.2. §3's threat model only excludes reverse engineering, JavaScript decompiling, and direct code analysis; it does not forbid an attacker from rewriting the prompt, cropping or inpainting the image, or using a second model to check the options. No such adaptive attacks are tested. An attacker who simply asks the model to ignore the decorative description and identify the hidden content is outside the tested evaluation, so the 100% figure does not support a general AI-hard claim.","section":"Abstract and §6.3, Table 5"},{"comment":"The 30 IllusionCAPTCHA test images are selected by generating 50 candidates and keeping the one with the lowest cosine similarity to the base image. This is a form of selection bias: the evaluation does not sample from the distribution of images the generator produces but from the extreme tail that is least similar to the base image. The reported 0% LLM success rate may therefore overstate the deception rate of a randomly generated instance. In addition, the illusion strength is a fixed free parameter (1.5) with no sensitivity analysis, so it is unclear how robust the result is to parameter changes.","section":"§5.1 and §6.2, Table 4"},{"comment":"The option design deliberately includes the exact prompt used to generate the illusion as a distractor, and another distractor is a verbose description of the illusionary element. The paper attributes the LLMs' 100% selection of the predicted option to visual illusions, but the evaluation does not ablate this option design. Without testing options with short, generic distractors or without the prompt-matching distractor, it remains possible that the LLMs are mostly responding to the length- and prompt-matching bias of the options rather than to the illusionary content. The contribution of the 'structured step-by-step method' is therefore not isolated from the confounding of option content.","section":"§5.2, §5.3, and §6.3, Table 5"},{"comment":"The human usability evidence is thin: RQ3 uses 10 participants and RQ6 uses 23, with no statistical testing or confidence intervals. Moreover, the RQ3 task (perceiving and describing an illusion) is not the same as the RQ6 task (selecting one of four multiple-choice options), so the 83%/88% visibility figures in Table 3 do not by themselves validate the 86.95% first-attempt pass rate in Table 6. The abstract's statement that IllusionCAPTCHA is 'outperforming other CAPTCHA systems' is not supported by any comparative user study in the paper, since no other CAPTCHA was presented to the same participants under the same protocol.","section":"§6.1 and §6.4, Tables 3 and 6"}],"minor_comments":[{"comment":"There are several typos and infelicities, e.g., 'capthchas' in the dataset collection paragraph, 'Overloaping' in Table 1, 'incoperate' in §5, and 'multimodel' in §5. The paper would benefit from a proofreading pass.","section":"§4.1"},{"comment":"The datasets and evaluation protocols are described at a high level, but it is not always clear how the multiple-choice options in RQ4 were presented to the LLMs (i.e., whether the model was given the four options or asked to answer openly). Clarifying the exact input format for each RQ would improve reproducibility.","section":"§4.1 and §6.2"},{"comment":"The paper states that 50 images are generated 'at a fixed illusion strength level of 1.5—an optimal value for human identification in this context,' but no evidence is provided for this optimality. A small sensitivity analysis or a reference to prior work would be useful.","section":"§5.1"},{"comment":"The discussion of adversarial attacks and cross-cultural adaptability is qualitative and does not include any experiments. The claim that adversarial images lack transferability is only supported by a reference, not by experiments on the proposed system.","section":"§7"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing to know: this paper has a new idea—using diffusion-generated visual illusions as CAPTCHAs—and it shows a plausible proof of concept. Under the authors' fixed prompt, GPT-4o and Gemini 1.5 Pro 2.0 fail all 30 examples, while 20 of 23 human participants pass on the first try. That is worth a serious look. The systematic survey of LLMs across CAPTCHA types in Section 4 is also useful, and the observation that humans and LLMs make similar errors on reasoning CAPTCHAs is a fair empirical point.\n\nWhere it gets soft: the '100% deception' headline in the abstract is not supported. RQ4/RQ5 test only two models, one inducement prompt (\"Tell us the true and detailed answer of this image\"), and a fixed four-option layout where the distractor is deliberately verbose and the correct answer is the hidden content. The threat model in Section 3 only rules out reverse engineering; it does not forbid an attacker from rewriting the prompt, cropping or inpainting the image, or using a second model to check the options. That is not a nitpick—it is the load-bearing condition for the central claim. The stress-test concern holds up on reading: an attacker who simply asks the model to ignore the decorative description and identify the hidden content is outside the tested setup, so the 100% figure is an artifact of the evaluation protocol, not a demonstrated property of the illusion mechanism.\n\nOther soft spots are smaller. The 30 test images were selected by lowest cosine similarity to the base image, which stacks the deck in favor of 'hard for AI' while making the human usability numbers less clean. The user studies have 10 and 23 participants, no control condition, and no released code, data, or prompts, so independent verification is not possible. The abstract's claim that IllusionCAPTCHA outperforms 'other CAPTCHA systems' rests on a comparison to the authors' own earlier user-study numbers, not a controlled head-to-head.\n\nThe citation pattern is fine. They build on their own Oedipus solver [7], and citing it as motivation is legitimate. There is no circular fitting; the free parameter (illusion strength) is fixed before evaluation.\n\nBottom line: this is a solid proof-of-concept paper that overclaims. A careful reader gets value from the survey and the new mechanism. It deserves a serious referee, but the referee should require the authors to narrow the claims to the tested setup and ideally to test against adaptive prompts before publication. My own verdict would be revise-and-resubmit, not reject outright.","headline":"A genuinely new CAPTCHA mechanism with a real proof-of-concept, but the headline '100% deception' claim is conditioned on a non-adversarial fixed prompt and a curated test set, so the paper overstates its security.","tokens_in":14940,"tokens_out":2006,"would_cite":false,"duration_ms":19303,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"IllusionCAPTCHA defeats LLMs 100% of the time while 86.95% of humans pass on the first attempt.","keywords":["CAPTCHAs","visual illusions","large language models","AI security","multimodal AI","Human-Easy AI-Hard","inducement prompt","illusion diffusion"],"falsifier":"Run the same 30 illusionary CAPTCHA images against GPT-4o or Gemini with a prompt such as 'Ignore the surface decoration; describe any hidden or embedded content, and then choose the option that matches the hidden content,' plus a simple crop or contrast adjustment, and measure the success rate; any rate above 0% would falsify the 100% deception claim.","tokens_in":13872,"feed_emoji":"🎭","tokens_out":6803,"duration_ms":57675,"temperature":0.7,"pith_summary":"The paper argues that current CAPTCHAs are no longer secure against multimodal large language models, and that reasoning-based CAPTCHAs are also hard for humans. It proposes IllusionCAPTCHA, which generates images whose surface appearance hides the true content from AI but remains readable to humans. The central claim is that under the tested interaction protocol, GPT-4o and Gemini 1.5 Pro 2.0 fail to solve the CAPTCHA 100% of the time, while 86.95% of human participants pass on the first attempt. If correct, this would give web services a CAPTCHA that is both harder for LLM-based bots and easier for people than current alternatives.","feed_headline":"IllusionCAPTCHA defeats LLMs 100% of the time","feed_subtitle":"A visual-illusion test that fools multimodal AI while 87% of humans pass on the first attempt.","key_machinery":"The load-bearing object is the illusion diffusion model (built on ControlNet) that fuses a base image with a user prompt, so the output image displays the prompt's content on its surface while preserving the base content underneath; the most confusing image is selected by lowest cosine similarity to the base. The second mechanism is the four-option multiple-choice frame with a deliberately verbose decoy describing the prompt's surface content, plus an inducement prompt ('Tell us the true and detailed answer of this image') that triggers LLM hallucination and length bias. Together they make the CAPTCHA a forced error for the tested LLMs and a quick recognition task for humans.","core_discovery":"The authors claim to have built the first illusion-based CAPTCHA that instantiates the 'Human-Easy but AI-Hard' paradigm: an image is blended with a decoy prompt so that the generated picture looks like the prompt to an AI vision system but still reveals the original content to a human. Four multiple-choice options are offered, and a step-by-step 'inducement prompt' instructs the model to give the true detailed answer, which exploits the models' tendency to produce long detailed descriptions and steers them to the wrong option. In evaluation, neither GPT-4o nor Gemini 1.5 Pro 2.0 solved any of the 30 illusionary CAPTCHAs under zero-shot or chain-of-thought prompting, and the inducement prompt made both models pick the intended wrong option in 100% of cases; 86.95% of human participants passed on the first attempt.","pith_inferences":["The 100% deception rate is an upper bound under the exact inducement prompt and four-option layout; an attacker who rewrites the prompt to ignore the surface description, or preprocesses the image, may defeat it, and that scenario is not tested.","The underlying mechanism suggests a benchmark: illusionary images could be used to measure whether a vision model truly perceives semantics or only matches surface text, a testable extension beyond CAPTCHAs.","An arms race is likely: fine-tuning on illusionary images or adding a second model to check options could close the gap, so the security claim may be time-limited."],"forward_implications":["If the result holds, existing LLM-based CAPTCHA solvers cannot pass IllusionCAPTCHA under the evaluated interaction, restoring a usable security layer for web services.","The inducement-prompt design becomes a template: any multimodal multiple-choice challenge can be structured to steer LLMs toward a predictable wrong answer.","Because 86.95% of humans pass on the first attempt, IllusionCAPTCHA would improve usability relative to reasoning-based CAPTCHAs, which the paper finds require a second attempt for most users.","The empirical finding that LLMs and humans make the same mistakes on existing CAPTCHAs implies that separate designs, not harder vision tasks, are needed to tell them apart."],"supporting_citations":[{"why":"The illusion generation tool used to blend a base image with a user prompt to produce the illusionary CAPTCHA images.","marker":"[2]"},{"why":"The conditional-control framework underlying the illusion diffusion model, enabling the generated image to keep the base content.","marker":"[31]"},{"why":"The prior LLM-based reasoning-CAPTCHA solver that this work extends and compares its security against.","marker":"[7]"},{"why":"One of the two evaluated multimodal LLMs, reported to solve none of the IllusionCAPTCHA instances.","marker":"[5]"},{"why":"The second evaluated multimodal LLM, also reported to solve none of the IllusionCAPTCHA instances.","marker":"[23]"},{"why":"The study showing LLMs produce long detailed image descriptions, the behavior the inducement prompt exploits.","marker":"[15]"},{"why":"The chain-of-thought prompting method used as an alternative inference strategy in the evaluation.","marker":"[28]"},{"why":"The prior empirical study of modern CAPTCHAs whose attempt-count findings are extended by this paper's user study.","marker":"[20]"}],"fun_headline_variants":["Visual illusion CAPTCHA stumps LLMs 100%, humans pass 87%","AI fails 100% on illusion CAPTCHA; 87% of humans pass first try","Illusion CAPTCHA: 100% AI failure, 87% human success rate","Illusion-based CAPTCHA: AI zero, humans 87% first try"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The 100% deception claim assumes the attacker uses the exact inducement prompt and the fixed four-option layout with no prompt rewriting and no image preprocessing; if an attacker simply asks the model to ignore the decorative text and identify hidden content, the reported rate has no support.","fun_headline_variants_meta":{"raw":{"variants":["Visual illusion CAPTCHA stumps LLMs 100%, humans pass 87%","AI fails 100% on illusion CAPTCHA; 87% of humans pass first try","Illusion CAPTCHA: 100% AI failure, 87% human success rate","Illusion-based CAPTCHA: AI zero, humans 87% first try"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001044,"raw_usage":{"total_tokens":4442,"prompt_tokens":1050,"completion_tokens":3392,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":666,"completion_tokens_details":{"reasoning_tokens":3299}},"tokens_in":666,"tokens_out":3392,"duration_ms":19974,"temperature":1.0,"reasoning_tokens":3299,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T19:14:49.045104+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 30 illusionary CAPTCHA images against GPT-4o or Gemini with a prompt such as 'Ignore the surface decoration; describe any hidden or embedded content, and then choose the option that matches the hidden content,' plus a simple crop or contrast adjustment, and measure the success rate; any rate above 0% would falsify the 100% deception claim.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The illusion generation tool used to blend a base image with a user prompt to produce the illusionary CAPTCHA images."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The conditional-control framework underlying the illusion diffusion model, enabling the generated image to keep the base content."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The prior LLM-based reasoning-CAPTCHA solver that this work extends and compares its security against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"One of the two evaluated multimodal LLMs, reported to solve none of the IllusionCAPTCHA instances."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The study showing LLMs produce long detailed image descriptions, the behavior the inducement prompt exploits."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The chain-of-thought prompting method used as an alternative inference strategy in the evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The prior empirical study of modern CAPTCHAs whose attempt-count findings are extended by this paper's user study."}],"review_version":1}