Pith. sign in

REVIEW 5 major objections 8 minor 28 references

Assessing Color Vision Test in Large Vision-language Models

T0 review · 5 major / 8 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper tries to establish that current large vision-language models score near chance on a purpose-built color vision test, and that targeted fine-tuning can lift accuracy to over 90%.

desk verdict Useful new benchmark showing LVLMs fail Ishihara-style color tests, but the fine-tuning claim is only same-distribution and needs OOD validation before it means general correction. read the letter →

arxiv 2507.11153 v1 pith:RZF2MPAJ submitted 2025-07-15 cs.CV cs.AI

Hongfei Ye , Bin Chen , Wenxi Liu , Yu Zhang , Zhao Li , Dandan Ni , Hongyang Chen This is my paper · ORCID
classification cs.CVcs.AI
keywords colorvisiontestlargevision-languagemodelsmodelevaluationsyntheticdatasetcolor-visionplatesLoRAfine-tuningvisualperceptionerroranalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that current large vision-language models lack robust color vision: on a new synthetic color vision test built in the style of standard human color-vision plates, the best of ten state-of-the-art models answers only about 21% of easy items and 18% of hard items correctly. It argues this failure is real and not captured by existing benchmarks, which devote under 1% of items to color. It further claims the deficit is largely correctable: LoRA fine-tuning of a 7B open model on 70% of the test data raises accuracy to 94% on the easy task and 92% on the hard task. If correct, color perception should be treated as a distinct, measurable capability in vision-language models rather than absorbed into general visual understanding.

What carries the argument

The load-bearing mechanism is the color vision test dataset itself: 5,450 synthetic plates in five content categories, generated to resemble standard human color-vision plates, filtered through a color-blindness simulator, and validated by three raters with low-agreement items discarded. The dataset serves as both the evaluation instrument—it produces the low baseline accuracies—and the training source for the LoRA fine-tuning experiment, where a 70/30 split yields the jump from roughly 16% to roughly 94%. The two task variants, CVTE with a category clue and CVTH without, together separate failures of color discrimination from failures of open-ended content identification.

What would settle it

Run the same CVTE and CVTH items on versions of the images converted to grayscale while preserving shape and layout; if models still score at or near the reported 15–21%, the benchmark is measuring shape recognition rather than color discrimination, and the central claim fails.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that color vision is a separable, currently unmet capability in large vision-language models. Ten models spanning open- and closed-source families were tested on a 5,450-image color vision dataset; the best easy-task score was 20.86% (JanusPro-7B) and the best hard-task score was 18.39% (GPT-4o), with the weakest model at 3.72%. The error taxonomy shows models usually answer with the right category but the wrong item (37.5% complete recognition errors) or say the content is unidentifiable (21.4%). LoRA fine-tuning of LLaVA1.5-7B on 70% of the dataset raised CVTE accuracy from 15.72% to 94.43% and CVTH accuracy from 11.31% to 92.23%, while full fine-tuning made the model refuse to answer. The paper reads these results as showing a trainable color-vision gap rather than an architectural impossibility.

Load-bearing premise

The evaluation rests on the assumption that the synthetic plates and their human-verified labels are a valid instrument for measuring color vision; if the images carry unintended cues or the labels are wrong, the reported accuracies and fine-tuning gains do not describe color perception.

Editorial extensions

If this is right

  • Color perception should be tracked as its own evaluation axis; current general benchmarks with under 1% color content would not reveal the near-chance performance documented here.
  • Deployers of vision-language models in color-sensitive settings such as autonomous driving or medical imaging should not assume human-level color discrimination without a dedicated test.
  • A modest fine-tuning intervention on synthetic color plates can close most of the measured gap, suggesting model improvement is accessible without full re-training.
  • GPT-4-based model scoring agrees more closely with human raters than the Meteor n-gram metric, so future color-vision evaluations should prefer judge-based or human scoring.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the same dataset could be used to probe where the failure lives—vision encoder versus language model—by comparing hue-versus-shape performance with the language generation head fixed.
  • Beyond the paper, the fine-tuning result likely reflects in-distribution learning on synthetic plates; a stricter test would fine-tune on synthetic data and evaluate on real-world color-coded photographs.
  • Beyond the paper, a grayscale control would tell whether the test measures color or shape: if grayscale versions still score near 20%, the benchmark is really about contour recognition.
  • Beyond the paper, the near-zero performance of the largest model (Qwen2.5VL-72B at 3.72%) hints that scale alone does not produce color discrimination, a pattern worth testing across model families.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 8 minor

Summary. The paper introduces a color vision testing benchmark for large vision-language models (LVLMs). The authors generate 5,450 synthetic color vision test images using the Daltonlens toolkit, filter them through Coblis, and organize them into five categories (numbers, animals, letters/Chinese characters, shapes, objects). They define two tasks: CVTE, which provides a category clue, and CVTH, which does not. Models are evaluated with METEOR, a GPT-4-based model score, and a human evaluation with triplicate runs. The reported results show that current LVLMs score around 20% or lower on both tasks, with JanusPro-7B at 20.86% on CVTE and GPT-4o at 18.39% on CVTH. The paper also analyzes error types and reports that LoRA fine-tuning of LLaVA1.5-7B improves accuracy to 94.43% on CVTE and 92.23% on CVTH using a 70/30 split of the same dataset.

Significance. If the benchmark is valid, this paper provides a useful resource for evaluating color perception in LVLMs and documents a striking deficiency: state-of-the-art models perform near or below 20% on standardized color vision tests, well below human performance. The two task variants and the error taxonomy are thoughtful, and the public data link supports reproducibility. However, the second half of the central claim, that targeted fine-tuning substantially corrects the deficiency, is not established by the current experiments because training and testing use the same synthetic distribution. The human-evaluation protocol also excludes low-agreement cases without reporting how many, and the claim that GPT-4 model scores align with human scores better than METEOR is not quantified. These gaps do not invalidate the benchmark itself, but they need to be addressed before the paper's conclusions can be fully supported.

major comments (5)
  1. [§4.6, Table 5] The fine-tuning experiment trains on 70% of the same Daltonlens-generated dataset and tests on the remaining 30%. Because the 5,450 images are derived from a limited set of templates and color schemes, high test accuracy (94.43% on CVTE, 92.23% on CVTH) may reflect memorization of image-level statistics rather than a generalized color-discrimination ability. The central claim that color vision deficiencies are 'correctable' requires out-of-distribution evaluation, for example on unseen plate designs, a different synthetic generator, altered color palettes, or natural images. Without such evidence, the abstract and conclusion overstate what the fine-tuning result demonstrates.
  2. [§3.3] The human-evaluation protocol states that scores with low inter-rater agreement (<0.5) are discarded and replaced, but the manuscript does not report how many cases were discarded, what the IRA values were, or which categories were affected. Since human scores are used to validate and compare against GPT-4 model scores, discarding disagreement removes exactly the cases where evaluation is most uncertain, potentially biasing the reported accuracies and the claimed alignment between model and human scores. The authors should report the number of discarded cases per category and provide a robustness analysis with and without the exclusion.
  3. [§4.3, Tables 2 and 3] All reported accuracies are averages of three trials, but no variance, confidence intervals, or per-trial results are provided. This is problematic for the main rankings: JanusPro-7B's 20.86% vs GPT-4o's 20.28% on CVTE and GPT-4o's 18.39% vs JanusPro-7B's 17.76% on CVTH are small margins that could easily be within run-to-run variability. Without error bars, the ranking claims and the interpretation of model strengths across categories are not statistically supported. Please report standard deviations or the full trial-level results.
  4. [§4.4, Q3, Table 4 and Figure 5] The paper claims that GPT-4 model scores 'are more closely aligned with human evaluation score compared to traditional machine metric,' but no quantitative measure of alignment is reported. METEOR values and model/human accuracies are on different scales, so visual inspection of Figure 5 does not establish a closer alignment. The authors should compute a correlation or agreement statistic (e.g., Pearson/Spearman correlation, Cohen's kappa) between METEOR and human scores and between GPT-4 model scores and human scores, and report whether the difference is meaningful or statistically significant.
  5. [§3.2] The validity of the benchmark depends on the generated images being legible and unambiguous for normal human vision. The manuscript states that Coblis was used to filter images and that a panel evaluated quality, but it does not report human legibility rates, the number of images rejected during the regeneration process, or any quantitative quality criteria. Coblis simulates color-deficient vision; it does not establish normal-view legibility or the absence of artifacts such as anti-aliasing, compression, or rendering errors. Reporting these numbers would substantially strengthen the claim that low LVLM accuracies reflect a color vision deficiency rather than a flawed test instrument.
minor comments (8)
  1. [Abstract and §1] The introduction says the evaluation 'revealed their outstanding capabilities in color vision test,' whereas the reported results show very poor performance (around 20%). This wording should be corrected to avoid contradiction.
  2. [§2] The benchmark name 'MMSTar' appears to be a typo for 'MMStar'.
  3. [Figure 5] The y-axis label reads 'Meteo' instead of 'Meteor'.
  4. [Table 2] The column labels are confusing: 'Met.' is defined as the weighted average of METEOR in each class, while 'Avg.' is the weighted average of model score; 'All' is used for the CVTH model score. Please use consistent and explicitly defined column names.
  5. [§4.4] The sentence 'These results suggests that these results highlight the differing strengths of each model' contains a duplicated phrase and a subject-verb agreement error; it should be rewritten.
  6. [Figures 2 and 4] The prompt format alternates between '{Content: ...}' and '{Contents: ...}' and between 'content' and 'contents.' Please unify the notation throughout the paper and in the released dataset.
  7. [§3.3] The IRA threshold of 0.5 is introduced without justification, and no IRA values are reported anywhere in the paper, despite the claim that IRA was computed.
  8. [§4.6] The fine-tuning section reports the average METEOR score for both CVTE and CVTH combined, but it is unclear how the two tasks are pooled. Please clarify the computation.

Circularity Check

0 steps flagged · score 0.0 of 10

No derivational circularity: the benchmark evaluation is independent of its own inputs, and the fine-tuning experiment is a held-out evaluation rather than a fitted prediction.

full rationale

The paper makes no first-principles derivation whose conclusion is fed back as an input. The benchmark images are generated from known ground-truth contents with the Daltonlens toolkit and independently quality-checked with Coblis and human raters; model scores are compared against these external annotations. The main empirical claims (low accuracies on CVTE/CVTH, error-type distribution, correlation with other benchmarks) are measurements, not tautologies. The only near-issue is the LoRA fine-tuning experiment in Section 4.6: the adapter is trained on 70% of the same Daltonlens-generated distribution and tested on the remaining 30%, so the reported 94.43%/92.23% gains demonstrate in-distribution learnability rather than generalized color-vision improvement. However, this is a held-out evaluation on the same benchmark and is not a fitted parameter renamed as a prediction; it is a validity and generalizability limitation, not circularity. Similarly, using GPT-4 as a judge while evaluating GPT-4o and GPT4V is a potential measurement bias, but the paper also reports human evaluation scores and finds close alignment, so the evaluator overlap is not load-bearing. No self-citation chain or imported uniqueness theorem is used. I therefore find no significant circularity.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The paper is an empirical benchmark study, not a derivation. It introduces no new physical entities. The central claims rest on domain assumptions about the validity of the generated test images and the evaluation protocols. The only hand-chosen numeric threshold that affects results is the IRA cutoff of 0.5 used to discard human evaluations.

free parameters (1)
  • IRA threshold = 0.5
    In Section 3.3, human evaluations with inter-rater agreement below 0.5 are discarded and replaced. This hand-chosen threshold affects which samples enter the human evaluation and is not justified by any analysis.
assumptions (3)
  • domain assumption The Ishihara-style color vision test is a valid measure of color vision ability.
    The paper motivates the task using the Ishihara test and assumes that constructed Ishihara-like images test color vision (Sections 1, 3.2).
  • domain assumption The Daltonlens toolkit and Coblis quality screening produce valid color vision test images with correct annotations.
    The dataset is generated and filtered with these tools, and the annotations are trusted as ground truth (Section 3.2).
  • domain assumption GPT-4 can reliably judge whether a model's response is correct relative to the ground truth.
    The model score uses GPT-4 as an evaluator (Sections 3.3, 4.3). If GPT-4 misjudges correctness, the reported accuracies change.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Assessing Color Vision Test in Large Vision-language Models." pith.science (2026). https://pith.science/paper/RZF2MPAJ

@misc{pith2026250711153,
  author       = {Pith},
  title        = {Pith review of: Assessing Color Vision Test in Large Vision-language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RZF2MPAJ}},
  note         = {Machine review of arXiv:2507.11153}
}
read the original abstract

With the widespread adoption of large vision-language models, the capacity for color vision in these models is crucial. However, the color vision abilities of large visual-language models have not yet been thoroughly explored. To address this gap, we define a color vision testing task for large vision-language models and construct a dataset \footnote{Anonymous Github Showing some of the data https://anonymous.4open.science/r/color-vision-test-dataset-3BCD} that covers multiple categories of test questions and tasks of varying difficulty levels. Furthermore, we analyze the types of errors made by large vision-language models and propose fine-tuning strategies to enhance their performance in color vision tests.

Figures

Figures reproduced from arXiv: 2507.11153 by the authors.

Figure 1
Figure 1. Comparison of previous study and standardized [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 3
Figure 3. By evaluating the responses, we can gauge the model’s [PITH_FULL_IMAGE:figures/full_fig_p002_3.png] view at source ↗
Figure 2
Figure 2. We present examples of our two tasks. The easy task includes cues that indicate the content category of the visual test [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figures from the paper (5 more)
Figure 3
Figure 3. Figure 3: Our setup for the color vision task. the quality of color vision test images. Coblis is a widely recognized tool for evaluating the quality of color vision test images in the field. It transforms any picture to simulate how it would appear to individuals with red, gree…
Figure 4
Figure 4. Figure 4: Color Vision Test Setting. Our proposed color vision test encompasses two task settings of varying difficulty. Participants [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 6
Figure 6. Figure 6: Correlation between other LVLMs benchmarks with [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Percentage of response error types 4.5 Error Attribution As shown in fig 7, the performance of the model across various recognition tasks can be summarized by several error categories: Incorrect Category Understanding (10.14%) occurs when the model fails to identify th…
Figure 8
Figure 8. Figure 8: Cases of responses from different LVLMs. [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 15 canonical work pages

  1. [1]

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond.arXiv preprint arXiv:2308.12966(2023)

    arXiv 2023

  2. [2]

    Marc G Berman, Michael C Hout, Omid Kardan, MaryCarol R Hunter, Grigori Yourganov, John M Henderson, Taylor Hanayik, Hossein Karimi, and John Jonides

    work page

  3. [3]

    Manojit Bhattacharya, Soumen Pal, Srijan Chatterjee, Sang-Soo Lee, and Chiranjib Chakraborty. 2024. Large language model to multimodal large language model: A journey to shape the biological macromolecules to biological sciences and medicine.Molecular Therapy-Nucleic Acids35, 3 (2024)

    work page 2024

  4. [4]

    Jennifer Birch. 1997. Efficiency of the Ishihara test for identifying red-green colour deficiency.Ophthalmic and Physiological Optics17, 5 (1997), 403–408

    work page 1997

  5. [5]

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al . 2024. Are We on the Right Way for Evaluating Large Vision-Language Models?arXiv preprint arXiv:2403.20330(2024)

    arXiv 2024

  6. [6]

    Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. 2025. Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling.arXiv preprint arXiv:2501.17811 (2025)

    arXiv 2025

  7. [7]

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al . 2024. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.arXiv preprint arXiv:2412.05271(2024)

    arXiv 2024

  8. [8]

    Can Cui, Yunsheng Ma, Xu Cao, Wenqian Ye, Yang Zhou, Kaizhao Liang, Jintai Chen, Juanwu Lu, Zichong Yang, Kuei-Da Liao, et al . 2024. A survey on mul- timodal large language models for autonomous driving. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 958–979

    work page 2024

Show all 28 references
  1. [9]

    Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, et al. 2024. Chatglm: A fam- ily of large language models from glm-130b to glm-4 all tools.arXiv preprint arXiv:2406.12793(2024)

    2024 arXiv

  2. [10]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card.arXiv preprint arXiv:2410.21276(2024)

    2024 arXiv

  3. [11]

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. 2023. Seed-bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125(2023)

    2023 arXiv

  4. [12]

    Guanzhen Li, Yuxi Xie, and Min-Yen Kan. 2024. MVP-Bench: Can Large Vision– Language Models Conduct Multi-level Visual Perception Like Humans?arXiv preprint arXiv:2410.04345(2024)

    2024 arXiv

  5. [13]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024. Visual instruc- tion tuning.Advances in neural information processing systems36 (2024)

    2024

  6. [14]

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al . 2024. Mmbench: Is your multi-modal model an all-around player?. InEuropean conference on computer vision. Springer, 216–233

    2024

  7. [15]

    Alex Melamud, Stephanie Hagstrom, and Elias Traboulsi. 2004. Color vision testing.Ophthalmic Genetics25, 3 (2004), 159–187

    2004

  8. [16]

    OpenAI. 2023. GPT-4V(ision) System Card. https://api.semanticscholar.org/ CorpusID:263218031

    2023

  9. [17]

    Galina V Paramei and David L Bimler. 2023. Color vision testing. InEncyclopedia of Color Science and Technology. Springer, 517–523

    2023

  10. [18]

    Romke Rouw, Stephen M Kosslyn, and Ronald Hamel. 1997. Detecting high-level and low-level properties in visual images and visual percepts.Cognition63, 2 (1997), 209–226

    1997

  11. [19]

    Ahnaf Mozib Samin, M Firoz Ahmed, and Md Mushtaq Shahriyar Rafee. 2024. ColorFoil: Investigating Color Blindness in Large Vision and Language Models. arXiv preprint arXiv:2405.11685(2024)

    2024 arXiv

  12. [20]

    WS Stiles. 1959. Color vision: the approach through increment-threshold sensi- tivity

    1959

  13. [21]

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530(2024)

    2024 arXiv

  14. [22]

    Qwen Team. 2025. Qwen2.5-VL. https://qwenlm.github.io/blog/qwen2.5-vl/

    2025

  15. [23]

    2013.Visual perception: An introduction

    Nicholas Wade and Mike Swanston. 2013.Visual perception: An introduction. Psychology Press

    2013

  16. [24]

    Jinge Wang, Qing Ye, Li Liu, Nancy Lan Guo, and Gangqing Hu. 2024. Scientific figures interpreted by ChatGPT: strengths in plot recognition and limits in color perception.NPJ Precision Oncology8, 1 (2024), 84

    2024

  17. [25]

    Zhenhua Xu, Yujia Zhang, Enze Xie, Zhen Zhao, Yong Guo, Kwan-Yee K Wong, Zhenguo Li, and Hengshuang Zhao. 2024. Drivegpt4: Interpretable end-to-end autonomous driving via large language model.IEEE Robotics and Automation Letters(2024)

    2024

  18. [26]

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. 2024. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. InProceedings of the IEEE/CVF Conference...

    2024

  19. [27]

    Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, et al. 2024. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems?. InEuropean Conference on Computer Vision. Springer, 169–186

    2024

  20. [2014]

    The perception of naturalness correlates with low-level visual features of environmental scenes.PloS one9, 12 (2014), e114572

    2014

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.