State-of-the-art vision-language models score only about 20% on a new synthetic color vision test, and LoRA fine-tuning on the same test distribution raises accuracy to about 94%.
ColorFoil: Investigating Color Blindness in Large Vision and Language Models
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
With the utilization of Transformer architecture, large Vision and Language (V&L) models have shown promising performance in even zero-shot settings. Several studies, however, indicate a lack of robustness of the models when dealing with complex linguistics and visual attributes. In this work, we introduce a novel V&L benchmark - ColorFoil, by creating color-related foils to assess the models' perception ability to detect colors like red, white, green, etc. We evaluate seven state-of-the-art V&L models including CLIP, ViLT, GroupViT, and BridgeTower, etc. in a zero-shot setting and present intriguing findings from the V&L models. The experimental evaluation indicates that ViLT and BridgeTower demonstrate much better color perception capabilities compared to CLIP and its variants and GroupViT. Moreover, CLIP-based models and GroupViT struggle to distinguish colors that are visually distinct to humans with normal color perception ability.
citation-role summary
background 1
citation-polarity summary
fields
cs.CV 1years
2025 1verdicts
CONDITIONAL 1roles
background 1polarities
background 1representative citing papers
citing papers explorer
-
Assessing Color Vision Test in Large Vision-language Models
State-of-the-art vision-language models score only about 20% on a new synthetic color vision test, and LoRA fine-tuning on the same test distribution raises accuracy to about 94%.