Pith. sign in

REVIEW 1 major objections 5 minor 29 references

Beyond Color Geometry: Evaluating Human-Like Color Representations in Vision Models

T0 review · 1 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash

Pith's one-line read This paper argues that masked-autoencoder vision models align with the graded, fuzzy way humans name colors more strongly than other transformer encoders, and that this difference survives once geometric color distance is controlled for.

desk verdict Useful new evaluation framework: fuzzy color reference plus geometry-controlled partial correlation; the MAE finding is striking, but the paper never says whether COLIBRI's training colors overlap the WCS evaluation chips—and no code/data is released. read the letter →

arxiv 2607.13647 v1 pith:YHVAGLUT submitted 2026-07-15 cs.CV cs.AI

classification cs.CVcs.AI
keywords colorrepresentationfuzzycategoriesmaskedautoencodersvisiontransformersrepresentationalsimilarityanalysishuman-modelalignmentCOLIBRIWorldSurvey
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Human color perception is graded: a color can belong partly to several named categories, yet most evaluations of vision models use geometric spaces or discrete labels that miss this structure. The paper evaluates eleven Vision Transformer encoders against COLIBRI, a fuzzy perceptual model with 86 overlapping categories fitted to human survey data, using three complementary measures: category boundaries, compactness, and graded alignment beyond what color geometry explains. It finds that category-level metrics barely separate the models, but graded alignment does: the two masked-autoencoder (MAE) models score highest, with non-overlapping confidence intervals on both measures. On natural images, MAE represents surface color globally while language-supervised models bind color to the foreground object, and layer-wise analysis shows MAE preserves graded structure toward the output. The upshot is that human-like color grounding has several distinct aspects and should not be reduced to a single score.

What carries the argument

The central reference object is COLIBRI, a fuzzy color space with 86 graded human-named categories defined by triangular/trapezoidal membership functions over Hue, Saturation, and Intensity, fitted to a 2,496-participant survey; each color receives an 86-dimensional membership vector and its dominant entry gives a categorical label. The evaluation combines representational similarity analysis with a first-order partial Spearman correlation that controls for the CIEDE2000/CIELAB geometric RDM, yielding pΔ, a measure of graded agreement beyond color geometry. This machinery is what lets the paper separate human category structure from ordinary perceptual distance.

What would settle it

Check the provenance of COLIBRI's training colors relative to the WCS chip set. If any of the 330 test chips (or near-duplicates) were used to fit the membership functions, recompute ρ_fuzzy and pΔ on a held-out Munsell palette; if the MAE advantage shrinks or vanishes, the paper's central claim is undermined. Alternatively, find a non-reconstruction encoder whose pΔ matches MAE when evaluated on held-out colors.

Watch

Extended reading notes

Core claim

Using COLIBRI's 86-dimensional fuzzy membership vectors as a human reference, the paper computes representational dissimilarity matrices for 330 World Color Survey chips rendered as flat patches and shaded spheres. MAE encoders reach ρ_fuzzy = 0.67 [0.65,0.70] and pΔ = 0.47 [0.43,0.50] on spheres, with bootstrap intervals that do not overlap the best non-MAE model (0.58 and 0.28); the same ordering holds on flat patches. Category-level metrics (ARI, silhouette) overlap broadly across all training regimes. Per-layer analysis shows MAE is the only regime that does not lose graded alignment toward the output, and on natural images MAE decodes background surface color most accurately while langu

Load-bearing premise

The load-bearing premise is that the COLIBRI fuzzy membership functions were fitted on colors that do not overlap the 330 World Color Survey chips used for evaluation; if the reference model was tuned on the test palette, the reported alignment would be partly circular.

Editorial extensions

If this is right

  • Color-grounding evaluations that stop at category labels or geometric distance can declare models similar even when their graded agreement with human color perception differs sharply.
  • Masked reconstruction appears to preserve low-level color structure during training, since MAE retains graded alignment toward the output layer whereas other regimes lose it.
  • Language-supervised training shapes color representations around foreground objects, not surface color in general; on isolated color patches these models align less with human fuzzy categories.
  • Human-like color grounding is multidimensional: category boundaries, compactness, and graded membership can diverge, so future benchmarks should report several measures rather than one.
  • The MAE advantage on flat patches shows the result does not depend on naturalistic rendering, extending to the simplest color stimuli.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If pixel-reconstruction pressure is the driver, then other reconstruction-based or low-level objectives (e.g., diffusion models, GAN reconstruction) should show similar graded color alignment; a controlled comparison across objectives would test this.
  • The WCS chips cover only 50 of COLIBRI's 86 categories; sampling a denser, broader set of colors would reveal whether the MAE lead holds for rare or boundary colors where fuzzy memberships matter most.
  • The paper does not establish that the COLIBRI training colors exclude the 330 WCS test chips; a provenance check or held-out color set is the key control before accepting the alignment numbers as unbiased.
  • The object-selectivity gap suggests color in language-supervised models is entangled with object identity; using object-masked image embeddings could separate surface-color coding from object-color coding.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

1 major / 5 minor

Summary. The paper proposes a framework for evaluating the color representations of Vision Transformer (ViT) encoders against COLIBRI, a fuzzy human color model with 86 graded categories fitted to survey data. The framework measures three complementary properties: category boundaries (silhouette), cluster compactness (ARI), and graded alignment beyond color geometry (ρ_fuzzy and pΔ, where pΔ is a partial correlation controlling for CIELAB distance). The evaluation uses 330 World Color Survey Munsell chips rendered as flat patches and shaded spheres, plus a natural-image probe from MegaCOIN. Across eleven encoders, the authors report that the two MAE models achieve the highest graded alignment on both flat and shaded stimuli, with non-overlapping bootstrap 95% confidence intervals (e.g., ρ_fuzzy = 0.67 [0.65,0.70] and pΔ = 0.47 [0.43,0.50] on shaded spheres for MAE-large). Layer-wise analysis shows MAE preserves this structure toward the output, while natural-image decoding indicates MAE encodes background color more globally and language-supervised models favor foreground-object color. The paper interprets these findings as evidence that color grounding has distinct aspects that should not be reduced to a single score.

Significance. If the result holds, the paper makes a useful contribution by introducing a reusable, parameter-free evaluation framework that separates graded category structure from geometric color similarity. The use of a fuzzy human reference is a step beyond discrete labels and continuous similarity judgments, and the geometry-controlled pΔ measure is a sensible way to isolate beyond-geometry alignment. The authors use public checkpoints and report bootstrap confidence intervals, which is good practice. The main claim—that MAE encoders exhibit the strongest graded human alignment—is potentially interesting and would add to evidence that training objectives shape human–model alignment. However, the validity of the entire empirical comparison rests on the independence of the COLIBRI reference from the 330 WCS chips used as test stimuli, and this independence is not established in the manuscript.

major comments (1)
  1. [III-A and III-B] The evaluation uses the 330 WCS Munsell chips as stimuli, while the COLIBRI reference is fitted to survey data from 2,496 participants. The manuscript does not state whether the survey stimuli used the same 330 chips or a different color palette. Because COLIBRI (ref. [9]) is co-authored by the same research group, this is a concrete risk of reference–test leakage. If the fitting set overlapped with the evaluation set, the reported ρ_fuzzy and pΔ values (Table II) would be inflated by construction, and the MAE advantage might not generalize to a reference fitted on disjoint colors. This is load-bearing for the central claim. I request a clear statement of the provenance of the COLIBRI fitting stimuli, and preferably a robustness check with a held-out set of colors (e.g., a subset of WCS chips or an independent color atlas).
minor comments (5)
  1. [Table II] The table header contains formatting errors: 'ALL ELEVEN ENCODERS,ORDERED BY SPHEREρ FUZZY .' has missing spaces. Also, in Section IV-A, 'MAE reaches ρ_fuzzy = 0.67' should specify MAE-large, since MAE-base has a slightly lower value (0.667) in Table II.
  2. [II, Table I] The last row of Table I reads 'Oursfuzzy human (COLIBRI)'—missing space before 'fuzzy'. Please fix the spacing for readability.
  3. [III-D-1] In Eq. (1), the notation ρ_m,f, ρ_m,l, and ρ_f,l is introduced, but it may help to explicitly define ρ_f,l as the Spearman correlation between the COLIBRI and CIELAB RDMs, and ρ_m,l as the model–CIELAB correlation, to avoid any ambiguity.
  4. [Fig. 5 / IV-C] The per-layer ρ_fuzzy values use mean-pooled token representations, whereas the final-embedding scores in Table II use different pooling (e.g., CLS token, post-projection embedding). The text acknowledges this, but the caption of Fig. 5 could reiterate that the absolute values are not directly comparable across pooling schemes.
  5. [General] The paper does not mention whether the evaluation code or data will be made available. Given the reproducibility of the framework, a statement about code release would be beneficial.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the evaluation pipeline is self-contained, with no fitted parameters in this paper and an external human-survey-derived reference.

full rationale

The paper's central measurements compare frozen model embeddings against COLIBRI membership vectors and MegaCOIN human color annotations. The RSA measures (rho_fuzzy and p_delta) are computed directly from RDMs via Spearman correlation and a first-order partial-correlation formula; no parameter is fitted to model outputs for the WCS analyses. The MAE advantage is an observed ranking with bootstrap confidence intervals, not a prediction derived from a fitted model. COLIBRI (ref. [9]) is co-authored by the same authors, but it is described as fitted to a survey of 2,496 participants and is not fitted to any of the eleven encoders, so its use as a human reference does not reduce to the paper's own inputs. The concern that COLIBRI's training colors might overlap with the 330 WCS evaluation chips would be an external-validity issue, but the paper provides no evidence of such overlap, and no equation or construction in the paper exhibits that reduction. Under the requirement to quote a specific circular step, none is present. The evaluation is therefore not circular, though the provenance of COLIBRI's fitting stimuli could merit additional disclosure.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The paper does not introduce free parameters or new entities; its central claim depends on the validity of the external COLIBRI reference, the (unstated) independence of that reference from the WCS test stimuli, the approximation behind the partial Spearman correlation, and the stimulus/probe assumptions.

assumptions (5)
  • domain assumption COLIBRI's fuzzy membership functions (86 categories over HSI) accurately represent human graded color perception.
    Used as the ground truth reference throughout; if these functions are not valid, all alignment measures are affected. Section III-B.
  • ad hoc to paper The 330 WCS Munsell chips used for evaluation are independent of the survey data used to fit COLIBRI.
    Assumed for non-circularity; the paper never states whether the same color stimuli were used in the human survey that produced COLIBRI. Section III-A and III-B.
  • standard math Applying the first-order partial correlation formula to Spearman rank correlations yields a valid measure of beyond-geometry alignment.
    Equation (1) uses the standard partial correlation formula on Spearman coefficients; this is a common approximation, though not a theoretically exact partial rank correlation.
  • domain assumption The shaded-sphere stimulus is an intermediate between uniform patches and natural images, preserving surface color while adding shading.
    The sphere rendering is used to make stimuli more naturalistic; if the shading changes the apparent color category, the chip-probe results could be confounded. Section III-A.
  • domain assumption Linear probe balanced accuracy on MegaCOIN foreground/background labels reflects the decodability of color terms in the encoder's representation, and the gap Δ reflects object-color selectivity.
    Used to draw conclusions about global vs. object-bound color representation. Section III-D2.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond Color Geometry: Evaluating Human-Like Color Representations in Vision Models." pith.science (2026). https://pith.science/paper/YHVAGLUT

@misc{pith2026260713647,
  author       = {Pith},
  title        = {Pith review of: Beyond Color Geometry: Evaluating Human-Like Color Representations in Vision Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YHVAGLUT}},
  note         = {Machine review of arXiv:2607.13647}
}
read the original abstract

Do vision models see colors the way humans do? Existing evaluations of color representations usually compare them with geometric spaces such as CIELAB or with discrete color labels. These references capture perceptual distance or category membership, but not the graded way in which people organize colors. We evaluate color grounding against a fuzzy perceptual model with 86 graded categories fitted to human survey data. The framework can be applied to any image encoder and measures three complementary properties: category boundaries, category compactness, and graded alignment beyond what color geometry alone can explain. Across eleven Vision Transformer encoders, the category-level results are broadly similar, whereas graded alignment differs substantially. Masked Autoencoders achieve the strongest beyond-geometry alignment, with confidence intervals that do not overlap those of the other encoders. A layer-wise analysis further shows that masked reconstruction preserves this structure toward the output. On natural images, MAE represents surface color globally, while language-supervised models encode color more strongly in relation to the foreground object. These results show that human-like color grounding has several distinct aspects that should not be reduced to a single score.

Figures

Figures reproduced from arXiv: 2607.13647 by the authors.

Figure 1
Figure 1. Overview of the evaluation pipeline. World Color Survey / Munsell chips are rendered as flat patches and shaded spheres, mapped to COLIBRI fuzzy [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Representative stimuli used in the experiments. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. COLIBRI category structure: nine hues, four saturation levels, three intensity levels. At very low saturation a color is named by an achromatic grey [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Natural images from Tiny ImageNet. (a) Balanced probe accuracy for background ( [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Agreement with graded human color structure, [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

29 extracted references · 5 linked inside Pith

  1. [9]

    Colibri fuzzy model: Color linguistic-based representation and interpretation,

    P. Shamoi, N. Toganas, M. Muratbekova, E. Kadyrgali, A. Yerkin, A. Igali, M. Ziyada, A. Adilova, A. Karatayev, and Y . Torekhan, “Colibri fuzzy model: Color linguistic-based representation and interpretation,” IEEE Access, vol. 13, pp. 205 932–205 956, 2025

  2. [1]

    A survey on vision transformer,

    K. Han, Y . Wang, H. Chen, X. Chen, J. Guo, Z. Liu, Y . Tang, A. Xiao, C. Xu, Y . Xu, Z. Yang, Y . Zhang, and D. Tao, “A survey on vision transformer,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 1, pp. 87–110, 2023

  3. [2]

    Vision-language models for vision tasks: A survey,

    J. Zhang, J. Huang, S. Jin, and S. Lu, “Vision-language models for vision tasks: A survey,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 8, pp. 5625–5644, 2024

  4. [3]

    Color categorization independent of color naming,

    K. Siuda-Krzywicka, C. Witzel, E. Chabani, M. Taga, C. Coste, N. Cools, S. Ferrieux, L. Cohen, T. Seidel Malkinson, and P. Bartolomeo, “Color categorization independent of color naming,”Cell Reports, vol. 28, no. 10, pp. 2471–2479.e5, 2019

  5. [4]

    Can language models encode perceptual structure without grounding? a case study in color,

    M. Abdou, A. Kulmizev, D. Hershcovich, S. Frank, E. Pavlick, and A. Søgaard, “Can language models encode perceptual structure without grounding? a case study in color,” inProceedings of the 25th Confer- ence on Computational Natural Language Learning, A. Bisazza and O. Abend, Eds. Online: Association for Computational Linguistics, Nov. 2021, pp. 109–132

  6. [5]

    Color models in image processing: a review and experimental comparison,

    M. Muratbekova, N. Toganas, A. Igali, M. Shagyrov, E. Kadyrgali, A. Yerkin, and P. Shamoi, “Color models in image processing: a review and experimental comparison,”Discover Applied Sciences, vol. 8, no. 5, Mar. 2026

  7. [6]

    Mapping language models to grounded con- ceptual spaces,

    R. Patel and E. Pavlick, “Mapping language models to grounded con- ceptual spaces,” inInternational conference on learning representations, 2022

  8. [7]

    Grounding the vector space of an octopus: Word meaning from raw text,

    A. Søgaard, “Grounding the vector space of an octopus: Word meaning from raw text,”Minds and Machines, vol. 33, no. 1, pp. 33–54, 2023

Show all 29 references
  1. [8]

    World color survey,

    P. Kay and R. S. Cook, “World color survey,” inEncyclopedia of color science and technology. Springer, 2023, pp. 1601–1607

  2. [10]

    Berlin and P

    B. Berlin and P. Kay,Basic Color Terms: Their Universality and Evolution. University of California Press, 1991

  3. [11]

    Efficient compression in color naming and its evolution,

    N. Zaslavsky, C. Kemp, T. Regier, and N. Tishby, “Efficient compression in color naming and its evolution,”Proceedings of the National Academy of Sciences, vol. 115, no. 31, pp. 7937–7942, 2018

  4. [12]

    Representational similarity analysis – connecting the branches of systems neuroscience,

    N. Kriegeskorte, “Representational similarity analysis – connecting the branches of systems neuroscience,”Frontiers in Systems Neuroscience, 2008

  5. [13]

    Getting aligned on representational alignment,

    I. Sucholutsky, L. Muttenthaler, A. Weller, A. Peng, A. Bobu, B. Kim, B. C. Love, E. Grant, I. Groen, J. Achterberget al., “Getting aligned on representational alignment,”arXiv preprint arXiv:2310.13018, 2023

  6. [14]

    Human alignment of neural network representations,

    L. Muttenthaler, J. Dippel, L. Linhardt, R. A. Vandermeulen, and S. Kornblith, “Human alignment of neural network representations,” arXiv preprint arXiv:2211.01201, 2022

  7. [15]

    A large-scale examination of inductive biases shaping high-level visual representation in brains and machines,

    C. Conwell, J. S. Prince, K. N. Kay, G. A. Alvarez, and T. Konkle, “A large-scale examination of inductive biases shaping high-level visual representation in brains and machines,”Nature communications, vol. 15, no. 1, p. 9383, 2024

  8. [16]

    Colorbench: Can vlms see and understand the colorful world? a comprehensive benchmark for color perception, reasoning, and robustness,

    Y . Liang, M. Li, C. Fan, Z. Li, D. Nguyen, K. Cobbina, S. Bhardwaj, J. Chen, F. Liu, and T. Zhou, “Colorbench: Can vlms see and understand the colorful world? a comprehensive benchmark for color perception, reasoning, and robustness,”Advances in Neural Information Processing ...

  9. [17]

    Do masked autoencoders learn a human- like geometry of neural representation? divergence and convergence across brains and machines during naturalistic vision,

    H. Karimi and S. Anzellotti, “Do masked autoencoders learn a human- like geometry of neural representation? divergence and convergence across brains and machines during naturalistic vision,” inNeurIPS 2025 Workshop on Symmetry and Geometry in Neural Representations, 2025

  10. [18]

    Exploring the categorical nature of colour perception: Insights from artificial networks,

    A. Akbarinia, “Exploring the categorical nature of colour perception: Insights from artificial networks,”Neural Networks, vol. 181, p. 106758, 2025

  11. [19]

    Systematic comparison of color representations between humans and deep neural networks: towards predicting human color perception in a vast color space,

    N. R. Wickramanayaka and M. Oizumi, “Systematic comparison of color representations between humans and deep neural networks: towards predicting human color perception in a vast color space,”bioRxiv, 2025, preprint

  12. [20]

    Parametric fuzzy sets for automatic color naming,

    R. Benavente, M. Vanrell, and R. Baldrich, “Parametric fuzzy sets for automatic color naming,”Journal of the Optical Society of America A, vol. 25, no. 10, pp. 2582–2593, 2008

  13. [21]

    The ciede2000 color-difference formula: Implementation notes, supplementary test data, and mathemat- ical observations,

    G. Sharma, W. Wu, and E. N. Dalal, “The ciede2000 color-difference formula: Implementation notes, supplementary test data, and mathemat- ical observations,”Color Research & Application, vol. 30, no. 1, pp. 21–30, 2005

  14. [22]

    Megacoin: enhancing medium-grained color perception for vision-language models,

    M.-C. Chiu, S. Wen, P.-Y . Chen, and X. Ma, “Megacoin: enhancing medium-grained color perception for vision-language models,” inPro- ceedings of the AAAI’26/IAAI’26/EAAI’26. AAAI Press, 2026

  15. [23]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clarket al., “Learning transferable visual models from natural language supervision,” inInternational conference on machine learning. PmLR, 2021, pp. 8748–8763

  16. [24]

    Reproducible scaling laws for contrastive language-image learning,

    M. Cherti, R. Beaumont, R. Wightman, M. Wortsman, G. Ilharco, C. Gordon, C. Schuhmann, L. Schmidt, and J. Jitsev, “Reproducible scaling laws for contrastive language-image learning,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023, pp....

  17. [25]

    Sigmoid loss for language image pre-training,

    X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer, “Sigmoid loss for language image pre-training,” inProceedings of the IEEE/CVF international conference on computer vision, 2023, pp. 11 975–11 986

  18. [26]

    Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features,

    M. Tschannen, A. Gritsenko, X. Wang, M. F. Naeem, I. Alabdul- mohsin, N. Parthasarathy, T. Evans, L. Beyer, Y . Xia, B. Mustafa et al., “Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features,”arXiv preprint arXiv...

  19. [27]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gellyet al., “An image is worth 16x16 words: Transformers for image recognition at scale,”arXiv preprint arXiv:2010.11929, 2020

  20. [28]

    Masked au- toencoders are scalable vision learners,

    K. He, X. Chen, S. Xie, Y . Li, P. Doll ´ar, and R. Girshick, “Masked au- toencoders are scalable vision learners,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 16 000–16 009

  21. [29]

    Dinov2: Learning robust visual features without supervision,

    M. Oquab, T. Darcet, T. Moutakanni, H. V o, M. Szafraniec, V . Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Noubyet al., “Dinov2: Learning robust visual features without supervision,”arXiv preprint arXiv:2304.07193, 2023

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.