{"id":"caddd140-56e3-4b7b-aec9-3304e510cabd","arxiv_id":"2502.04470","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"CLIP shows a robust achromatic color bias and a strong text-over-vision Stroop effect, with color-selective neurons appearing in shallow layers.","lead":"CLIP, a widely used vision-language model, labels colors in images well for chromatic colors but systematically fails to treat white, gray, and black as colors, and it reads written color words instead of perceiving font colors. The paper documents these failures with large synthetic experiments and analyzes internal neurons to suggest why.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"All headline color-deficiency results rely on top-1 cosine similarity against a fixed set of color-word prompts; the achromatic failure could be a text-encoder/prompt artifact rather than a visual encoding deficiency.","rationale":"The reader's weakest assumption is the same one I consider load-bearing: the paper defines 'what CLIP sees' as the argmax of cosine similarity between one image embedding and a fixed set of color-word text embeddings. I agree with that identification, and I sharpen it with a concrete visual-side control. The reason this matters more than the missing statistical uncertainty or unquantified neuron thresholds is that it targets the core semantic content of the claim: if the representation actually contains achromatic color information, then the paper's stated conclusion that CLIP's color understanding 'systematically fails for achromatic stimuli' is incorrect or at best about the text/alignment branch; if the representation lacks it, the conclusion stands. The Stroop/text-priority results are more robust because the written word is physically present, but they are about a different behavior (reading vs. perceiving) and do not by themselves establish an achromatic visual deficiency. I do not think the paper should be rejected: the empirical patterns are plausible and the paper makes a good-faith effort to test multiple prompts. The conditional verdict is appropriate, with the proposed linear-probe control as the key condition. A secondary reproducibility issue—the paper never specifies which CLIP checkpoint was used—reinforces the need for the test to be run on a named model.","tokens_in":7142,"tokens_out":9017,"duration_ms":95981,"concrete_test":"Recreate a held-out subset of the Section 2 dataset (achromatic object on chromatic background and chromatic object on achromatic background). Train a linear probe on the CLIP image embeddings (the paper does not say which CLIP checkpoint, so fix one, e.g., RN50) to predict the object's color label from the image embedding alone, and evaluate on held-out images. If the probe classifies the achromatic object color with accuracy comparable to chromatic-object accuracy (e.g., >90%), then the image encoder does encode achromatic color, and the near-0% top-1 text-prompt result in Experiment 2 is a text-encoder/prompt artifact rather than a visual color deficiency. This directly separates the visual representation from the prompt-ranking operationalization.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that 'CLIP does not attribute the color word to achromatic parts of the image in presence of a Chromatic color' (Section 2, Experiment 2) is inferred solely from the ordering of 11 color-name text prompts by cosine similarity to the full image embedding. This operationalization conflates three separable factors: what the image encoder represents, what the text encoder's color-word embeddings encode, and how the contrastive alignment ranks those embeddings. A gray object beside a red background can fail the text-prompt ranking even if the image encoder encodes its grayness well, because the text encoder may carry a strong prior that 'The color of the object is red' is a more probable caption than '...is gray'. The paper varies prompt templates (Experiment 3) but never varies the output channel: all answers are text-label rankings, so no measurement isolates the visual color representation. The Stroop finding is less vulnerable to this objection because the written word is physically present in the image, but the achromatic deficiency—a core part of the central claim—is underdetermined. A visual-side control is therefore needed before accepting that CLIP's color understanding 'systematically fails for achromatic stimuli.'","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents a set of experiments on synthetic images and Stroop-style stimuli to evaluate CLIP's ability to label colors. The authors report that CLIP correctly labels chromatic colors but fails to assign achromatic labels (white, gray, black) when a chromatic color is present in the image, and that CLIP strongly prefers the written color word over the actual font color in Stroop tasks. To explain these behaviors, the authors introduce a Color-Label Selectivity Index and analyze neuron activations across CLIP's visual encoder, proposing a taxonomy of neuron types and reporting the presence of color multimodal neurons in shallow layers. The main contributions are empirical: a systematic behavioral characterization of CLIP's color biases and a descriptive neuron-level analysis.","tokens_in":7420,"tokens_out":7774,"duration_ms":70797,"significance":"If confirmed, the behavioral findings are valuable for the VLM community, as they highlight systematic color deficiencies in a widely used model. The use of large synthetic datasets with multiple prompt templates is a methodological strength, and the Stroop experiments provide a compelling demonstration of text-over-vision bias. The neuron analysis is exploratory and less central; its contribution would be strengthened by more rigorous definitions and quantitative validation. The paper does not provide code or model specifications, which limits its immediate reproducibility.","major_comments":[{"comment":"The central claim that \"CLIP does not attribute the color word to achromatic parts of the image in presence of a Chromatic color\" is inferred exclusively from the ranking of 11 color-name text prompts by cosine similarity to the full image embedding. This operationalization conflates the image encoder's representation, the text encoder's embedding of color words, and the contrastive alignment. The paper varies the prompt template (Experiment 3) but never varies the output channel; all answers are text-label rankings. A visual-side control—for example, a linear probe trained on the image encoder's features to predict the actual object/background color, or a forced-choice task with achromatic labels in a matched context—is needed to determine whether the achromatic failure reflects a visual encoding deficiency or a text-encoder prior that achromatic words are less plausible color labels. Without such a control, the causal interpretation of the achromatic deficiency (and the hypothesis in the Conclusions about achromatic backgrounds in datasets) is underdetermined.","section":"Section 2, Experiments 1-2 (Table 2)"},{"comment":"The neuron taxonomy is defined using unquantified thresholds. The criteria \"high Color-Label Selectivity Index\", \"high activation for any word\", \"does not reach 50% of the maximum activation\", and the conjunction required for \"Color Multimodal\" are not specified numerically (e.g., percentile cutoffs, minimum activation ratios, or how the three color-label conditions are combined). The reported distribution of neuron types and the finding of shallow-layer color multimodal neurons cannot be reproduced or evaluated without these thresholds.","section":"Section 4, \"Activation Analysis\" and Figure 5"},{"comment":"The paper does not identify the exact CLIP model variant or implementation used for any experiment. The Introduction says the image encoder is \"Resnet-like\" and the neuron analysis refers to \"blocks of convolutional layers\", but the model name (e.g., RN50, RN101, ViT-B/32), the pretraining dataset, and the checkpoint or codebase are not given. All experimental results, and especially the neuron-level analysis, depend on this specificity.","section":"Throughout (all experiments)"},{"comment":"The word \"significant\" is used without supporting statistics. The paper reports aggregate percentages over 500 samples per condition but provides no error bars, confidence intervals, or significance tests. This is particularly problematic for the claim that achromatic backgrounds have \"significant lower errors\" in the Stroop test, where the effect appears to be smaller than the main text-over-vision bias. At minimum, the authors should report per-condition variability and avoid \"significant\" without a test.","section":"Abstract and Section 3 (Experiment 5)"},{"comment":"The definition of the Color-Label Selectivity Index is unclear. The notation uses N_c as a count of images but then sums activation values w_{j,i,L} over j, and the denominator sums over N_l w_{l,i,L}. The text should clarify whether f_c is a count ratio or an activation-weighted frequency, and should specify the set of images (\"N cropped top scoring images\") over which the index is computed (e.g., which dataset, how crops are selected, how many). This is necessary to interpret the subsequent neuron classification.","section":"Section 4, Equation (1)"}],"minor_comments":[{"comment":"The text says \"the performance decreases to 0.19%\" when the object is achromatic, but Table 2 shows 0.15% for the corresponding cell; please align the text with the table.","section":"Section 2, Experiment 2"},{"comment":"The text says \"2.23%\" for the font-color percentage of the selected prompt, but Table 3 shows 2.35%; please correct.","section":"Section 3, Experiment 3"},{"comment":"The abstract uses \"VML\" while the rest of the paper uses \"VLM\"; please standardize.","section":"Abstract"},{"comment":"Reference [5] contains \"Akash Ghosh1 et al.\" with a stray superscript; please clean up the citation.","section":"Introduction"},{"comment":"Please provide a reproducibility statement or release the code and data; the paper's value would be greatly increased by including the synthetic dataset generation code and the neuron analysis scripts.","section":"General"},{"comment":"In the description of the neuron categories, the paper should explicitly define what \"high activation\" means in terms of the activation distribution (e.g., relative to the top-scoring ImageNet patches).","section":"Section 4, neuron categories"}],"recommendation":"major_revision","confidential_remarks":"The paper is an empirical study of CLIP's color behavior with a descriptive neuron-level analysis. The behavioral experiments are large and clearly presented, but the interpretation overreaches by attributing the achromatic failure to visual encoding without controlling for the text-encoder prior. The neuron taxonomy needs quantitative thresholds and model specification before it can be considered reproducible. The paper would be suitable for a vision or multimodal venue after these points are addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The main news here is that CLIP really does have a reproducible achromatic blind spot: when an achromatic object or background sits next to a chromatic one, CLIP almost always labels the chromatic stimulus, and the text-over-vision Stroop effect is severe, especially with colored backgrounds. Those findings are supported by large synthetic datasets, multiple prompt templates, and consistent percentages across conditions. The neuron analysis is also a genuine addition—particularly the observation of color multimodal neurons in shallow layers, which extends the usual deep-layer story from prior work. The new Color-Label Selectivity Index is a reasonable tool, and the hue-selectivity correlation with ImageNet (R=0.965) is a nice sanity check.\n\nThat said, the paper has real soft spots. There are no error bars or significance tests despite the word “significant” being used; the thresholds for neuron categories are unquantified; and no code or data are released, which makes the claims harder to verify. The stress-test worry about the prompt-ranking operationalization is worth taking seriously, but it is not fatal: the same text prompts identify chromatic objects correctly, so a pure text-encoder prior against achromatic labels would not explain the asymmetry. Still, a visual-side control (e.g., a linear probe on image embeddings) would make the central claim much tighter. The neuron taxonomy is defined on the same Stroop data used to report the taxonomy, which is a mild circularity, though the behavioral results stand independently. The proposed explanation—that achromatic colors are prevalent as backgrounds in training data—is a plausible hypothesis but not tested.\n\nWho should read this: anyone working on CLIP interpretability, color understanding in multimodal models, or evaluating CLIP for downstream tasks where color matters. It is not a methodological breakthrough, but it documents a real deficiency that practitioners should know about. I would send it to peer review with a request for revision: add error bars and data/code, quantify the neuron thresholds, and address the operationalization concern with at least one probe experiment. The core empirical findings deserve to be in the literature, and the neuron-level analysis is a useful starting point for follow-up work.","headline":"The paper's behavioral evidence for CLIP's achromatic bias and text-priority is solid and worth engaging with, while the neuron-level mechanistic claims are more speculative and need sharper quantification.","tokens_in":7914,"tokens_out":2591,"would_cite":true,"duration_ms":29232,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CLIP understands chromatic colors but treats white, gray, and black as non-colors and prefers reading color words over perceiving font colors, a deficiency traceable to text-selective neurons.","keywords":["CLIP","color naming","achromatic colors","Stroop effect","multi-modal neurons","neuron selectivity","visual-language models"],"falsifier":"Use the same colored-text images but present the answer options as color swatches (patches of color) instead of color-name words, and measure accuracy on achromatic font colors and font-color identification. If CLIP still fails on achromatic colors and still returns the written color name, the deficiency is visual; if accuracy rises sharply, the reported failures are an artifact of prompting with color names.","tokens_in":6971,"feed_emoji":"🎨","tokens_out":7427,"duration_ms":64581,"temperature":0.7,"pith_summary":"This paper asks whether CLIP, the contrastive visual-language model used widely for zero-shot tasks, genuinely understands color as a visual attribute. Using synthetic images of colored shapes and a Stroop-style dataset of color words rendered in colored fonts, the authors show that CLIP names chromatic colors accurately but systematically refuses achromatic labels: white, gray, and black are almost never assigned when a chromatic color is present. They also demonstrate a strong reading bias: asked for the font color, CLIP outputs the written color name in 81.1% of white-background cases and only 2.35% of cases when a colored background is added. At the neuron level, the paper attributes these failures to a high number of text-selective neurons in deep layers and a smaller set of color multi-modal neurons in shallow layers. If the paper is right, any downstream system built on CLIP embeddings carries a built-in color deficiency: it cannot treat achromatic colors as colors, and it reads color words instead of seeing colors.","feed_headline":"CLIP reads color words instead of seeing font colors","feed_subtitle":"Asked the font color, it answers the written color name 81% of the time; on colored backgrounds only 2.35% get the font color right.","key_machinery":"The evaluation machinery is a zero-shot color-naming task: for each image, a set of color-label text prompts is ranked by cosine similarity between the CLIP text and image embeddings, and the highest-scoring label is the prediction. This turns every experiment into a direct comparison of how the text encoder and image encoder align. For the internal analysis, the paper introduces a Color-Label Selectivity Index, $f_c(n_{i,L}) = \\frac{\\sum_{j=1}^{N_c} w_{j,i,L}}{\\sum_{l=1}^{N} w_{l,i,L}}$, which measures how often a specific color label appears among the top-activating images for a neuron. This index is used to classify neurons into five types—Color, Any Word, Color Word, Color Multimodal, and Not activated—and to map their distribution across layers.","core_discovery":"On the paper's own terms, the central discovery is that CLIP's color ability is split: it correctly links chromatic color labels to visual stimuli, but it does not treat achromatic stimuli as belonging to the color concept. In the shape experiments, when the asked object is achromatic and the background chromatic, correct object-color prediction falls to 0.19%; when the background is achromatic and the object chromatic, correct background-color prediction falls to 5%. In the Stroop experiments, CLIP's font-color accuracy is 16.7% on a white background and 2.35% on a colored background, while the written color name is chosen 81.1% and 59.5% of the time respectively. The neuron analysis then shows why: deep layers are dominated by text-selective neurons, and the color multi-modal neurons that do exist are few and appear in shallow layers.","pith_inferences":["This suggests a testable extension: if the same Stroop images are evaluated with color swatches instead of color-name prompts as answer options, and achromatic accuracy stays near zero, the failure is in the visual encoder; if accuracy jumps, the reported metric was partly a text artifact.","The shallow-layer color multi-modal neurons suggest that low-level attribute concepts are encoded earlier in CLIP than previously assumed; probing these neurons might predict a model's color robustness before full evaluation.","Models finetuned from CLIP for image generation or editing may inherit the achromatic deficiency, producing black, gray, or white objects only with difficulty; this could be checked by prompting such models for achromatic objects in colorful scenes.","The authors' progressive-training hypothesis could be tested directly by re-training a small CLIP variant on an ordered curriculum and measuring achromatic Stroop accuracy before and after."],"forward_implications":["CLIP-based zero-shot color classifiers will systematically mislabel white, gray, and black whenever a chromatic color is in view, regardless of which image region the prompt asks about.","Any image that contains a color word will be labeled by reading rather than by perceived color, so downstream tasks such as font-color extraction or color-aware captioning will fail on such images.","The reading bias grows with visual distractions: adding a colored background drops font-color accuracy from 16.7% to 2.35%.","Since text-selective neurons concentrate in deep layers while color multi-modal neurons are few and shallow, the deficiency is structural rather than a simple threshold artifact, and will not be cured by prompt rewording alone."],"supporting_citations":[{"why":"Supplies the CLIP model and the contrastive text-image embedding space used in every experiment.","marker":"[10]"},{"why":"Documents multimodal neurons in CLIP and first reports a Stroop-like reading bias, which this paper extends with quantitative datasets.","marker":"[6]"},{"why":"Defines the Stroop task that Experiments 3-5 adapt to test whether CLIP reads color words instead of perceiving font colors.","marker":"[14]"},{"why":"Provides the color selectivity index used to measure how color-dependent each visual-encoder neuron is.","marker":"[11]"},{"why":"Proposes the class selectivity index that the paper adapts into its new Color-Label Selectivity Index.","marker":"[12]"},{"why":"Provides the ImageNet dataset on which top-scoring patches are found to compute neuron color selectivity.","marker":"[4]"},{"why":"Supplies the 11 basic color terms that define the label vocabulary and the synthetic dataset colors.","marker":"[2]"},{"why":"Shows text understanding improves CLIP's color categorization, motivating the comparison of text versus visual color information.","marker":"[1]"}],"fun_headline_variants":["CLIP fails achromatic colors, over-prioritizes text","CLIP's Stroop effect: written color beats font color","CLIP has color blind spots for white, gray, black","CLIP chooses color names over actual font colors","Neuron study: CLIP's text bias harms color vision"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results assume that picking the top-scoring color-label prompt by cosine similarity faithfully measures what CLIP perceives as the image's color, so a bias in how the prompt text is phrased could masquerade as a visual deficiency.","fun_headline_variants_meta":{"raw":{"variants":["CLIP fails achromatic colors, over-prioritizes text","CLIP's Stroop effect: written color beats font color","CLIP has color blind spots for white, gray, black","CLIP chooses color names over actual font colors","Neuron study: CLIP's text bias harms color vision"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00061,"raw_usage":{"total_tokens":2843,"prompt_tokens":949,"completion_tokens":1894,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":565,"completion_tokens_details":{"reasoning_tokens":1810}},"tokens_in":565,"tokens_out":1894,"duration_ms":13678,"temperature":1.0,"reasoning_tokens":1810,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T22:36:03.199741+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Use the same colored-text images but present the answer options as color swatches (patches of color) instead of color-name words, and measure accuracy on achromatic font colors and font-color identification. If CLIP still fails on achromatic colors and still returns the written color name, the deficiency is visual; if accuracy rises sharply, the reported failures are an artifact of prompting with color names.","supporting_citations":[{"cited_title":"Multimodal neurons in artificial neural net- works","cited_arxiv_id":null,"evidence_quote":"Documents multimodal neurons in CLIP and first reports a Stroop-like reading bias, which this paper extends with quantitative datasets."},{"cited_title":"Studies of interference in serial ver- bal reactions","cited_arxiv_id":null,"evidence_quote":"Defines the Stroop task that Experiments 3-5 adapt to test whether CLIP reads color words instead of perceiving font colors."},{"cited_title":"Color encoding in biologically-inspired convolutional neural networks","cited_arxiv_id":null,"evidence_quote":"Provides the color selectivity index used to measure how color-dependent each visual-encoder neuron is."},{"cited_title":"Understanding trained cnns by indexing neuron se- lectivity","cited_arxiv_id":null,"evidence_quote":"Proposes the class selectivity index that the paper adapts into its new Color-Label Selectivity Index."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the ImageNet dataset on which top-scoring patches are found to compute neuron color selectivity."},{"cited_title":"Basic Color Terms: Their Uni- versality and Evolution","cited_arxiv_id":null,"evidence_quote":"Supplies the 11 basic color terms that define the label vocabulary and the synthetic dataset colors."},{"cited_title":"Exploring the categorical nature of colour perception: Insights from artificial networks","cited_arxiv_id":null,"evidence_quote":"Shows text understanding improves CLIP's color categorization, motivating the comparison of text versus visual color information."}],"review_version":1}