{"id":"c9ab4eac-cfd1-4731-af50-edf5ccbc06cc","arxiv_id":"2412.06184","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A new 19,000-image photorealistic color illusion dataset shows that vision-language models are partially fooled by color illusions, producing human-aligned answers more often than chance but rarely matching human perception.","lead":"The paper creates a 19,000-image photorealistic dataset of color illusions (contrast, stripe, and filter types) and tests vision-language models on color comparison questions framed as either based on pixel values or based on human perception.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The unverified pixel-preservation of ControlNet-generated images is load-bearing: if target regions drift in color, pixel-based ground truth and deception rates are invalid.","rationale":"The reader's weakest assumption correctly identifies the load-bearing unverified premise: ControlNet-generated images must preserve the pixel-level color relationships of the procedural sources for the pixel-based evaluation labels to be valid. I agree with this assessment and do not find a different, more fundamental flaw. The concern is concrete and directly testable: the authors' released code permits regenerating the images and comparing region means against the intended labels. If the check passes, the central results stand; if it fails, the reported deception rates and the claimed drop in accuracy on illusion images are not reliably interpretable. The paper still has independent value: the RCID dataset, the human validation procedure, and the controlled evaluation framework are worthwhile contributions, and the qualitative finding that fine-tuned VLMs are vulnerable to these images is plausible. However, because the pixel-grounding issue is not verified anywhere in the manuscript and directly underpins the quantitative claims, a fully accept verdict would be premature. The reader's CONDITIONAL verdict remains appropriate, with the condition being verification of pixel preservation and correction of the abstract's overstatement if needed.","tokens_in":13552,"tokens_out":5923,"duration_ms":64519,"concrete_test":"Sample at least 50 contrast and 50 stripe images from the final RCID test set, along with their source procedural images and generation parameters. Segment the two target regions (using known coordinates for the procedural source and the released code for the generated images) and compute per-region mean RGB and CIELAB values. For images labeled \"same but look different,\" require mean Delta E_76 <= 2 and per-channel mean difference <= 3 after ControlNet generation; for \"different but look same,\" require the intended difference to remain in the correct direction. Then recompute the No-Illusion and Deception rates using the actual measured pixel relationships instead of the intended labels. If more than 10% of sampled images violate the intended pixel relationship, the pixel-based metrics and the central accuracy-drop claim require re-evaluation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that VLMs exhibit human-like perceptual biases on color illusions rests on pixel-level ground-truth labels: for contrast and stripe images labeled \"same but look different,\" the two target regions must have identical mean pixel colors, and for \"different but look same\" they must differ in the intended direction. Section 3.1 never verifies that the ControlNet-generated realistic images preserve these relationships. The pipeline description is also internally inconsistent: training conditions on a 10x10 quantized color grid G (x0 = G in the loss), while the generating stage says x0 = S and produces I' = sum_t (x_t + epsilon_theta), which is not a standard diffusion denoising update. Because diffusion generation with text conditioning and random noise can shift local colors, the final images may not satisfy the intended pixel comparisons. If one region is shifted more than another, the \"pixel values\" answers used for No-Illusion and Deception rates in Section 4 are wrong, so the reported >75% non-illusion accuracy and the accuracy drop on illusion images could reflect mislabeled pixel comparisons rather than genuine perceptual bias. Human validation confirms only perceived illusion/deception, not exact RGB preservation, so it cannot rescue the pixel-based metrics. The abstract's stronger claim of human similarity is additionally unsupported for base models that perform near chance without fine-tuning, but the pixel-preservation issue is the more fundamental threat.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an automated pipeline for generating photorealistic color-illusion images and introduces RCID, a dataset of 19,000 images spanning contrast, stripe, and filter illusions, with human validation labels. The authors evaluate several open-source VLMs after fine-tuning on non-illusion images, reporting strong accuracy on non-illusion images but significantly lower accuracy on illusion images, with a substantial fraction of human-like responses. They also investigate the roles of the visual encoder and linguistic/commonsense priors, and propose a mixed-training method that lets a model answer according to either pixel values or human perception.","tokens_in":13786,"tokens_out":2580,"duration_ms":27575,"significance":"If the results hold, this is a valuable contribution: RCID is much larger and more realistic than prior illusion datasets, the human-annotation protocol is independent and the fine-tuning design helps separate color-comparison ability from illusion susceptibility. The paper also goes beyond benchmark reporting by studying model size, prompt effects, and the visual/linguistic sources of bias. The code and data are promised open-source, which strengthens reproducibility. The main risk is that the pixel-level ground truth of the generated images is never verified; this threatens the validity of the no-illusion/deception metrics and, in turn, the central claim of human-like perceptual bias.","major_comments":[{"comment":"The generation stage as written is not a standard diffusion process and is inconsistent with the training objective. Equation (1) trains the model to denoise from x0 = G (the quantized image), but the generating-stage formula I' = sum_t (x_t + epsilon_theta(...)) sums noisy samples and predicted noise across all time steps. This is not the usual recursive denoising update and it is unclear what x_t denotes after the first step. Because the paper's central claim depends on the final images preserving the intended color relationships, the actual generation algorithm must be specified precisely and reproducibly.","section":"3.1"},{"comment":"The pixel-value ground truth used for No-Illusion and Deception rates in Section 4 rests on the assumption that the ControlNet-generated images I' preserve the mean-color relationships of the procedural images S (e.g., the two target squares have identical pixel colors in 'same but look different' contrast illusions, and differ in the intended direction in 'different but look same' cases). This is never verified after generation. Human validation in Section 3.4 only confirms that participants were deceived; it does not confirm that the final images satisfy the intended pixel comparisons. ControlNet is conditioned on text and noise, so local color shifts are plausible; if one target region shifts more than another, the reported >75% non-illusion accuracy and the accuracy drop on illusion images could partly be an artifact of mislabeled pixel comparisons rather than genuine perceptual bias. The authors should measure the target-region mean colors in the final images, filter or re-label images that fail the intended comparisons, and rerun the affected metrics.","section":"3.1 and 4.1"},{"comment":"The abstract's claim that 'all studied VLMs exhibit perceptual biases similar human vision' is stronger than the evidence in the paper. Appendix B.4 shows that without fine-tuning, models perform near random guessing on contrast and stripe tasks even on non-illusion images, so their 'human-like' behavior on illusions is not distinguishable from chance at the base-model level. The human-like bias is convincingly demonstrated only for models after fine-tuning on non-illusion images. The claim should be qualified to the fine-tuned setting, or the base-model analysis should be used to support a weaker claim about susceptibility rather than similarity to human perception.","section":"Abstract and Appendix B.4"}],"minor_comments":[{"comment":"There are typos: 'similar human vision' should be 'similar to human vision' in the abstract; 'an simple training method' in Section 1; 'fileds' in Section 6.","section":"Abstract and 6"},{"comment":"The notation is inconsistent: x0 is defined as the quantized grid G in the training stage but as the procedural image S in the generating stage, and the symbol C is used both for colors and for text captions. Please use distinct symbols.","section":"3.1"},{"comment":"The analysis in Figure 10 reports correlations between color descriptor diversity and pixel-value accuracy, but the figure would benefit from error bars or a clearer statement of the number of color terms per language and the significance of the correlation.","section":"5.2"}],"recommendation":"major_revision","confidential_remarks":"The pixel-preservation issue is fixable within the scope of the paper: the authors can post-verify the generated images and either filter the dataset or re-label the pixel ground truth. I recommend major revision rather than rejection, but the authors should address the generation-formula inconsistency and qualify the abstract's claim. The human-data collection appears careful, and the dataset has clear potential value to the community."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the RCID dataset is the contribution, and it's a good one. 19k photorealistic illusion images across contrast/stripe/filter families, human validation from 241 Prolific participants, control sets, open code and data. That alone justifies referee time.\n\nWhat the paper does well: the control setup is thoughtful. Fine-tuning on non-illusion images to establish a color-comparison baseline, then measuring the drop on illusion images, is the right way to separate illusion susceptibility from basic inability. The experiments span a broad set of open VLMs and include pure-vision models, which strengthens the claim that the bias is at least partly perceptual rather than purely linguistic. The finding that fine-tuned models hit >75% on non-illusion but drop sharply on illusion images is consistent with prior internet-sourced studies and is the paper's most solid result.\n\nThe soft spots are real, and one is load-bearing. The abstract says all studied VLMs exhibit perceptual biases similar to human vision, but the appendix shows that without fine-tuning, models are near chance on contrast/stripe even for non-illusion images. The similarity only emerges after fine-tuning. That's still interesting, but it's not 'similar to human vision' in any unqualified sense. More importantly, Section 3.1 never verifies that ControlNet preserves the exact pixel relationships from the procedural source. For 'same but look different' images, the pixel ground truth is 'exactly same' only if the two target regions have identical mean RGB in the final image. ControlNet with text conditioning and random noise can shift local colors; human validation only records whether the illusion is perceived, not whether the RGB values survived. The generating-stage equation I' = sum_t (x_t + eps_theta) is also not a standard diffusion denoising update, which suggests either a typo or a pipeline issue. If pixel drift is nontrivial, the No-Illusion and Deception rates are computed against mislabeled ground truth. This is fixable: sample final images, measure region means, report drift, and either filter or re-derive labels. The mechanism analyses (color distance, stripe counts, language frequency) are correlational and lack significance tests, but that is a minor issue compared to the pixel check.\n\nBottom line: this is a dataset paper, and the dataset is worth having. The human-similarity claim needs to be scaled back, and the pixel-preservation check is necessary before the quantitative metrics can be trusted. If the authors do that, the paper is a solid contribution. I'd send it to peer review with major revision requested, and I'd cite it for the dataset.","headline":"RCID is a genuinely useful dataset, but the pixel-preservation assumption is load-bearing and unverified, so the human-similarity claim needs softening.","tokens_in":14371,"tokens_out":2467,"would_cite":true,"duration_ms":24125,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Current vision-language models perceive color illusions much as humans do, a new 19,000-image photorealistic dataset shows—and they can be trained to report both human-like and pixel-based judgments.","keywords":["color illusions","vision-language models","perceptual bias","photorealistic dataset","contrast illusion","stripe illusion","filter illusion","instruction tuning"],"falsifier":"Compare, on a random sample of RCID contrast and stripe images, the mean RGB values of the two regions named in the question. If a nontrivial fraction of images labeled 'same' actually have different mean pixel colors (or vice versa), the pixel-based ground truth is broken and the reported deception rates would need to be recomputed. One could also re-run the VLM evaluation on the procedural source images versus the ControlNet-realistic versions: if the human-like response rate changes substantially, the photorealistic rendering step is introducing confounds.","tokens_in":13291,"feed_emoji":"👁️","tokens_out":7036,"duration_ms":60798,"temperature":0.7,"pith_summary":"The paper asks whether vision-language models (VLMs) are susceptible to color illusions the way humans are, and proposes a way to test this at scale. It introduces an automated pipeline that turns simple, procedurally generated illusion images into photorealistic scenes, producing RCID, a dataset of 19,000 images spanning contrast, stripe, and filter illusions, half of which contain an illusion and half of which are controls. Each image is paired with two questions: one asking about color according to pixel values, the other according to human perception. After fine-tuning on non-illusion images, all studied VLMs score high on non-illusion images but drop markedly on illusion images, and a substantial share of their wrong answers match human perception rather than pixel values. The paper concludes that current VLMs inherit human-like perceptual biases for color illusions, and that a simple mixed-training baseline can make a model report either human-perceived or pixel-based colors on request.","feed_headline":"Color illusions fool vision-language models, 19k-image test finds","feed_subtitle":"A 19,000-image dataset shows current models mirror human perception and can be taught to tell it apart from pixel truth.","key_machinery":"The mechanism that carries the argument is a three-stage dataset-generation pipeline combined with paired-question evaluation. Simple illusion images are first generated procedurally (backgrounds with different brightness, stripes with alternating colors, etc.), then a conditional diffusion model maps them to photorealistic scenes while retaining the intended color relationships; filter illusions are made by shifting hue in HSV space on MS COCO images. For every image, GPT-4o generates a natural-language question asking about color differences or object color, and two prompts are attached: one asks for the judgment 'based on pixel values' and the other 'based on human perception.' Human validation on Prolific decides which images truly deceive at least 3 of 5 viewers. This design lets the paper label each image with two ground truths—pixel truth and human truth—so a model response can be classified as accurate, human-like, or neither.","core_discovery":"The central claim is that current vision-language models exhibit perceptual biases similar to human vision when confronted with realistic color illusions. The evidence comes from RCID: models fine-tuned to answer color-comparison questions reach over 75% accuracy on non-illusion images, while accuracy on illusion images is significantly lower, and a portion of the errors are 'human-like'—they agree with human judgments rather than with pixel values. The paper also reports that model size increases human-like responding, that external prompts such as chain-of-thought or few-shot examples shift responses only modestly, and that fine-tuning on illusion images can push models toward either pixel-based or human-like answers. A mixed-training baseline lets a single model produce the correct pixel-based answer under a 'pixel values' prompt and the correct human-based answer under a 'human perception' prompt.","pith_inferences":["If the same pattern holds more broadly, color-illusion susceptibility could serve as a probe for how much a VLM's judgments rely on low-level visual processing versus stored world knowledge, complementing other perceptual benchmarks.","One testable extension is to vary the photorealistic generation process (e.g., different diffusion models or strengths of realism) and check whether human-like response rates track the fidelity of the scene, which would separate the contribution of image realism from the model's internal priors.","The RCID-style paired-question setup could be adapted to other perceptual constancies, such as lightness, size, or shape illusions, to see whether 'human-like' errors are a general property of VLM visual encoders or specific to color."],"forward_implications":["Because VLMs show human-like biases on photorealistic illusions, downstream applications that rely on color judgments in real images will inherit those systematic errors.","The success of mixed instruction tuning shows that models can be trained to report pixel-based or human-perception answers on demand, giving application developers a lever to choose the appropriate behavior.","The small effect of chain-of-thought and few-shot prompting indicates that these biases are not superficial output artifacts but are rooted in visual representations and learned priors.","The positive correlation between model size and human-like responses suggests that scaling model size alone will not eliminate these perceptual biases."],"supporting_citations":[{"why":"Supplies the conditional diffusion method that turns procedural illusion images into photorealistic scenes, the core of dataset generation.","marker":"[41]"},{"why":"Provides the GVIL metrics (human-like rate, no-illusion rate) used to score VLM responses on illusion images.","marker":"[42]"},{"why":"IllusionVQA, the prior web-sourced illusion dataset whose memorization problem motivates the need for a fresh realistic dataset.","marker":"[33]"},{"why":"The LLaVA-1.5 architecture used as the base model for the mixed-training baseline and many of the experiments.","marker":"[21]"},{"why":"Shows VLMs' limited color-comparison ability, which motivates the fine-tuning step that isolates illusion susceptibility from general color-judgment ability.","marker":"[12]"}],"fun_headline_variants":["VLMs exhibit human-like biases on color illusions","19,000 illusion images show VLMs see color like humans","Color illusions reveal VLMs mirror human perception","Vision-language models exhibit human-like color biases"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The photorealistic images produced by the conditional diffusion model preserve the exact pixel-level color relationships of the procedurally generated source images, so that the 'based on pixel values' ground-truth labels are correct for every illusion image.","fun_headline_variants_meta":{"raw":{"variants":["VLMs exhibit human-like biases on color illusions","19,000 illusion images show VLMs see color like humans","Color illusions reveal VLMs mirror human perception","Vision-language models exhibit human-like color biases"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000623,"raw_usage":{"total_tokens":2823,"prompt_tokens":823,"completion_tokens":2000,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":439,"completion_tokens_details":{"reasoning_tokens":1949}},"tokens_in":439,"tokens_out":2000,"duration_ms":14323,"temperature":1.0,"reasoning_tokens":1949,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:55:33.186192+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare, on a random sample of RCID contrast and stripe images, the mean RGB values of the two regions named in the question. If a nontrivial fraction of images labeled 'same' actually have different mean pixel colors (or vice versa), the pixel-based ground truth is broken and the reported deception rates would need to be recomputed. One could also re-run the VLM evaluation on the procedural source images versus the ControlNet-realistic versions: if the human-like response rate changes substantially, the photorealistic rendering step is introducing confounds.","supporting_citations":[{"cited_title":"role”: “system, “content","cited_arxiv_id":null,"evidence_quote":"Provides the GVIL metrics (human-like rate, no-illusion rate) used to score VLM responses on illusion images."},{"cited_title":"Visual instruction tuning, 2023","cited_arxiv_id":null,"evidence_quote":"The LLaVA-1.5 architecture used as the base model for the mixed-training baseline and many of the experiments."},{"cited_title":"Smith, Wei-Chiu Ma, and Ranjay Krishna","cited_arxiv_id":null,"evidence_quote":"Shows VLMs' limited color-comparison ability, which motivates the fine-tuning step that isolates illusion susceptibility from general color-judgment ability."}],"review_version":1}