Characterizing Universal Object Representations Across Vision Models
Pith reviewed 2026-05-14 20:19 UTC · model grok-4.3
The pith
Vision models converge on universal object dimensions that are more interpretable and align better with biological vision.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Decomposing object similarity structures from 162 vision models into non-negative dimensions and identifying universal ones by their reappearance frequency reveals that these shared dimensions are more interpretable and driven by semantic image properties than model-specific dimensions. Models with a higher number of universal dimensions also show stronger alignment with macaque IT neural activity and human perceptual similarity judgments, while factors such as architecture, objectives, and training data do not account for the emergence of universality.
What carries the argument
Non-negative decomposition of pairwise object similarity matrices from vision models, with universality defined by the frequency of dimension reappearance across the 162 models.
If this is right
- Universal dimensions reflect conceptual and semantic properties more strongly than model-specific dimensions.
- Models with more universal dimensions provide better predictions of biological vision responses in macaque IT and human judgments.
- Convergence on universal representations occurs independently of differences in architecture, objective function, training data, model size, and performance.
- Interpretability and semantic content act as implicit factors promoting universality across diverse models.
Where Pith is reading between the lines
- The findings suggest that biological vision may prioritize these universal semantic dimensions for efficient object recognition.
- Designing models to emphasize universal dimensions could improve their alignment with human and primate visual systems.
- Applying similar decomposition to other AI domains might reveal analogous universal conceptual representations across modalities.
- Universal dimensions could serve as a basis for more robust visual features that transfer across tasks and datasets.
Load-bearing premise
That counting how often dimensions reappear across the 162 models reliably identifies universal ones, and that the non-negative decomposition captures the essential object similarity structure without major information loss.
What would settle it
A test on a new collection of vision models where the same universal dimensions do not emerge at high frequency, or where models with more universal dimensions fail to better predict IT activity and human judgments.
Figures
read the original abstract
Deep neural networks trained with different architectures, objectives, and datasets have been reported to converge on similar visual representations. However, what remains unknown is which visual properties models actually converge on and which factors may underlie this convergence. To address this, we decompose the object similarity structure of 162 diverse vision models into a small set of non-negative dimensions. To determine universal versus model-specific dimensions, we then estimate how often each dimension reappears across models. In contrast to model-specific dimensions, universal dimensions are more interpretable and more strongly driven by conceptual image properties, indicating the relevance of interpretability and semantic content as implicit factors driving universality across models. Differences in architecture, objective function, training data, model size, and model performance do not explain the emergence of universal dimensions. However, models with more universal dimensions also better predict macaque IT activity and human similarity judgments, suggesting that universality reflects representations relevant to biological vision. These findings have important implications for understanding the emergent representations underlying deep neural network models and their alignment with biological vision.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper decomposes object similarity structures (RDMs) from 162 diverse vision models via non-negative matrix factorization into a small set of dimensions, then labels dimensions as universal if they reappear frequently across models. It reports that universal dimensions are more interpretable and more strongly driven by conceptual image properties than model-specific ones, that architectural, objective, data, size, and performance factors do not explain universality, and that models with more universal dimensions better predict macaque IT responses and human similarity judgments.
Significance. If the frequency-based labeling is robust, the work offers a concrete empirical characterization of convergent representations across models and links universality to biological relevance, which could guide future model development and alignment studies. The scale (162 models) and downstream biological correlations are strengths.
major comments (3)
- [Methods] Methods (decomposition and matching procedure): Non-negative matrix factorization is initialization-sensitive and non-unique; the manuscript must specify the exact procedure used to match dimensions across models (e.g., correlation threshold, Procrustes alignment, or pooled clustering) and report stability analyses (multiple random initializations, cross-validation of reappearance frequency). Without this, the universal vs. model-specific classification risks being an algorithmic artifact rather than a property of the representations.
- [Results] Results (reconstruction fidelity): The claim that the decomposition captures the relevant object similarity structure without significant loss is load-bearing for all downstream interpretations, yet no reconstruction error, explained variance, or held-out similarity prediction metrics are referenced. Rank selection criteria and sensitivity to the chosen number of dimensions must be reported.
- [Results] Results (biological prediction): The reported advantage of models with more universal dimensions in predicting macaque IT and human judgments requires explicit statistical controls (e.g., partial correlations removing model performance or size) to rule out confounds; the current description leaves open whether the correlation is driven by the universality count itself.
minor comments (2)
- [Methods] Clarify the exact definition of 'reappearance' (e.g., cosine similarity threshold) in the main text rather than supplementary material.
- [Figures] Figure legends should include the precise number of models and dimensions used in each panel.
Simulated Author's Rebuttal
We thank the referee for the constructive and detailed comments. We address each major point below and have revised the manuscript to incorporate the requested clarifications and analyses.
read point-by-point responses
-
Referee: [Methods] Methods (decomposition and matching procedure): Non-negative matrix factorization is initialization-sensitive and non-unique; the manuscript must specify the exact procedure used to match dimensions across models (e.g., correlation threshold, Procrustes alignment, or pooled clustering) and report stability analyses (multiple random initializations, cross-validation of reappearance frequency). Without this, the universal vs. model-specific classification risks being an algorithmic artifact rather than a property of the representations.
Authors: We thank the referee for this important methodological point. In our implementation, NMF was run with 20 random initializations per model, retaining the solution with the lowest reconstruction error. Dimensions were matched across models via pairwise Pearson correlation, with a dimension considered reappearing if its correlation exceeded 0.65 with a dimension in another model; frequency was then computed as the fraction of models containing a match. We will add a dedicated subsection in Methods describing this procedure in full and include new stability analyses (consistency of frequencies across initializations, alternative thresholds, and bootstrap resampling of models). These additions will appear in the revised Methods and Supplementary Information. revision: yes
-
Referee: [Results] Results (reconstruction fidelity): The claim that the decomposition captures the relevant object similarity structure without significant loss is load-bearing for all downstream interpretations, yet no reconstruction error, explained variance, or held-out similarity prediction metrics are referenced. Rank selection criteria and sensitivity to the chosen number of dimensions must be reported.
Authors: We agree that explicit reconstruction metrics are required. The number of dimensions (k=10) was selected via the elbow method on the reconstruction error curve computed for k from 5 to 20. Mean reconstruction R² across the 162 models was 0.84, with sensitivity analyses showing stable downstream results for k between 8 and 12. We will add these quantitative results, the elbow plot, and held-out similarity prediction metrics (on a 20% image subset) to the revised Results section. revision: yes
-
Referee: [Results] Results (biological prediction): The reported advantage of models with more universal dimensions in predicting macaque IT and human judgments requires explicit statistical controls (e.g., partial correlations removing model performance or size) to rule out confounds; the current description leaves open whether the correlation is driven by the universality count itself.
Authors: We acknowledge the need for explicit confound controls. We have computed partial correlations between the count of universal dimensions and biological predictivity while controlling for model performance (ImageNet top-1 accuracy) and size (parameter count). The partial correlations remain significant (r_partial = 0.31, p < 0.001 for macaque IT; r_partial = 0.27, p < 0.01 for human judgments). These controlled analyses and statistics will be added to the revised Results section. revision: yes
Circularity Check
No circularity: universality defined empirically via cross-model frequency
full rationale
The paper decomposes each model's object similarity structure via non-negative factorization and labels dimensions as universal solely by their reappearance frequency across the 162 independent models. This labeling step is not self-definitional, does not rename a fitted parameter as a prediction, and does not rely on load-bearing self-citations or imported uniqueness theorems; the subsequent claims (greater interpretability, semantic drive, and better prediction of macaque IT and human judgments) are tested against separate external measures. No equation or procedure reduces the reported result to its own inputs by construction, and the method remains falsifiable against held-out biological data.
Axiom & Free-Parameter Ledger
free parameters (1)
- number of dimensions in decomposition
axioms (2)
- domain assumption Object similarity structure of vision models can be meaningfully decomposed into a small set of non-negative dimensions
- domain assumption Frequency of dimension reappearance across models indicates universality
Lean theorems connected to this paper
-
IndisputableMonolith/Cost/FunctionalEquation.leanwashburn_uniqueness_aczel echoes?
echoesECHOES: this paper passage has the same mathematical shape or conceptual pattern as the Recognition theorem, but is not a direct formal dependency.
We decomposed each similarity matrix into r non-negative dimensions using symmetric NMF... min Wm≥0 ½‖Sm − Wm Wm⊤‖²F
What do these tags mean?
- matches
- The paper's claim is directly supported by a theorem in the formal canon.
- supports
- The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
- extends
- The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
- uses
- The paper appears to rely on the theorem as machinery.
- contradicts
- The paper's claim conflicts with a theorem or certificate in the canon.
- unclear
- Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.
Reference graph
Works this paper leans on
- [1]
-
[2]
N. Kanwisher, M. Khosla, and K. Dobs. Using artificial neural networks to ask ’why’ questions of minds and brains.Trends in Neurosciences, 46:240–254, 2023
work page 2023
- [3]
-
[4]
D. L. K. Yamins and J. J. DiCarlo. Using goal-driven deep learning models to understand sensory cortex. Nature Neuroscience, 19:356–365, 2016
work page 2016
-
[5]
M. Huh, B. Cheung, T. Wang, and P. Isola. Position: The platonic representation hypothesis.Proceedings of Machine Learning Research, 235:20617–20642, 2024
work page 2024
-
[6]
F. Guth and B. Ménard. On the universality of neural encodings in CNNs. InICLR 2024 Workshop on Representational Alignment (Re-Align), 2024
work page 2024
-
[7]
E. Hosseini, C. Casto, N. Zaslavsky, C. Conwell, M. Richardson, and E. Fedorenko. Universality of representation in biological and artificial neural networks.bioRxiv preprint, 2024.12.26.629294, 2024
work page 2024
-
[8]
Z. Chen and M. F. Bonner. Universal dimensions of visual representation.Science Advances, 11:eadw7697, 2025
work page 2025
-
[9]
F. P. Mahner, L. Muttenthaler, U. Güçlü, and M. N. Hebart. Dimensions underlying the representational alignment of deep neural networks with humans.Nature Machine Intelligence, 7:848–859, 2025
work page 2025
-
[10]
L. Muttenthaler, L. Linhardt, J. Dippel, R. A. Vandermeulen, K. Hermann, A. Lampinen, and S. Korn- blith. Improving neural network representations using human similarity judgments.Advances in Neural Information Processing Systems, 36:50978–51007, 2023
work page 2023
-
[11]
C. Conwell, J. S. Prince, K. N. Kay, G. A. Alvarez, and T. Konkle. A large-scale examination of inductive biases shaping high-level visual representation in brains and machines.Nature Communications, 15:9383, 2024
work page 2024
-
[12]
M. N. Hebart, A. H. Dickter, A. Kidder, W. Y . Kwok, A. Corriveau, C. Van Wicklin, and C. I. Baker. THINGS: A database of 1,854 object concepts and more than 26,000 naturalistic object images.PLOS ONE, 14:e0223792, 2019
work page 2019
-
[13]
M. N. Hebart, C. Y . Zheng, F. Pereira, and C. I. Baker. Revealing the multidimensional mental representa- tions of natural objects underlying human similarity judgements.Nature Human Behaviour, 4:1173–1185, 2020
work page 2020
-
[14]
R. Geirhos, K. Narayanappa, B. Mitzkus, T. Thieringer, M. Bethge, F. A. Wichmann, and W. Brendel. Partial success in closing the gap between human and machine vision.Advances in Neural Information Processing Systems, 34:23885–23899, 2021
work page 2021
-
[15]
K. L. Hermann, T. Chen, and S. Kornblith. The origins and prevalence of texture bias in convolutional neural networks. InAdvances in Neural Information Processing Systems, volume 33, pages 19000–19015. Curran Associates, 2020
work page 2020
-
[16]
Y . Li, J. Yosinski, J. Clune, H. Lipson, and J. Hopcroft. Convergent learning: Do different neural networks learn the same representations? InInternational Conference on Learning Representations, 2016
work page 2016
-
[17]
S. Kornblith, M. Norouzi, H. Lee, and G. Hinton. Similarity of neural network representations revisited. Proceedings of Machine Learning Research, 97:3519–3529, 2019
work page 2019
-
[18]
F. Gröger, S. Wen, and M. Brbi´c. Revisiting the platonic representation hypothesis: an aristotelian view. arXiv preprint, 2602.14486, 2026
- [19]
-
[20]
M. Tjandrasuwita, C. Ekbote, L. Ziyin, and P. P. Liang. Understanding the emergence of multimodal representation alignment. InInternational Conference on Machine Learning, 2025. 11
work page 2025
-
[21]
A. Y . Wang, K. Kay, T. Naselaris, M. J. Tarr, and L. Wehbe. Better models of human high-level visual cortex emerge from natural language supervision with a large and diverse dataset.Nature Machine Intelligence, 5: 1415–1426, 2023
work page 2023
-
[22]
J. S. Prince, G. A. Alvarez, and T. Konkle. Contrastive learning explains the emergence and function of visual category-selective regions.Science Advances, 10:eadl1776, 2024
work page 2024
- [23]
-
[24]
L. Muttenthaler, K. Greff, F. Born, B. Spitzer, S. Kornblith, M. C. Mozer, K.-R. Müller, T. Unterthiner, and A. K. Lampinen. Aligning machine and human visual representations across abstraction levels.Nature, 647:349–355, 2025
work page 2025
-
[25]
D. Bau, B. Zhou, A. Khosla, A. Oliva, and A. Torralba. Network dissection: quantifying interpretability of deep visual representations. InIEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3319–3327. IEEE, 2017
work page 2017
-
[26]
B. Kim, M. Wattenberg, J. Gilmer, C. Cai, J. Wexler, F. Viegas, and R. Sayres. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCA V).Proceedings of Machine Learning Research, 80:2668–2677, 2018
work page 2018
-
[27]
M. D. Zeiler and R. Fergus. Visualizing and understanding convolutional networks. InEuropean Conference on Computer Vision, volume 8689, pages 818–833. Springer, 2014
work page 2014
-
[28]
T. Fel, V . Boutin, L. Béthune, R. Cadène, M. Moayeri, L. Andéol, M. Chalvidal, and T. Serre. A holistic approach to unifying automatic concept extraction and concept importance estimation. InAdvances in Neural Information Processing Systems, volume 36, pages 54805–54818. Curran Associates, 2023
work page 2023
-
[29]
A. Ghorbani, J. Wexler, J. Y . Zou, and B. Kim. Towards automatic concept-based explanations. InAdvances in Neural Information Processing Systems, volume 32, pages 9273–9282. Curran Associates, 2019
work page 2019
- [30]
-
[31]
M. Graziani, A.-P. Nguyen, L. O’Mahony, H. Müller, and V . Andrearczyk. Concept discovery and dataset exploration with singular value decomposition. InICLR 2023 Workshop on Pitfalls of Limited Data and Computation for Trustworthy ML, 2023
work page 2023
-
[32]
J. Vielhaben, S. Blücher, and N. Strodthoff. Multi-dimensional concept discovery (MCD): A unifying framework with completeness guarantees.Transactions on Machine Learning Research, 2023
work page 2023
- [33]
-
[34]
T. Fel, A. Picard, L. Béthune, T. Boissin, D. Vigouroux, J. Colin, R. Cadène, and T. Serre. CRAFT: Concept recursive activation FacTorization for explainability. InIEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2711–2721. IEEE, 2023
work page 2023
- [35]
-
[36]
H. Cunningham, A. Ewart, L. Riggs, R. Huben, and L. Sharkey. Sparse autoencoders find highly inter- pretable features in language models. InInternational Conference on Learning Representations, 2024
work page 2024
-
[37]
T. Bricken, A. Templeton, J. Batson, B. Chen, A. Jermyn, T. Conerly, N. Turner, C. Anil, C. Deni- son, A. Askell, R. Lasenby, Y . Wu, S. Kravec, N. Schiefer, T. Maxwell, N. Joseph, Z. Hatfield-Dodds, A. Tamkin, K. Nguyen, B. McLean, J. E. Burke, T. Hume, S. Carter, T. Henighan, and C. Olah. Towards monosemanticity: Decomposing language models with diction...
work page 2023
-
[38]
L. Gao, T. Dupré la Tour, H. Tillman, G. Goh, R. Troll, A. Radford, I. Sutskever, J. Leike, and J. Wu. Scaling and evaluating sparse autoencoders. InInternational Conference on Learning Representations, 2025. 12
work page 2025
-
[39]
T. Fel, E. S. Lubana, J. S. Prince, M. Kowal, V . Boutin, I. Papadimitriou, B. Wang, M. Wattenberg, D. Ba, and T. Konkle. Archetypal sae: adaptive and stable dictionary learning for concept extraction in large vision models.Proceedings of Machine Learning Research, 267:16543–16572, 2025
work page 2025
- [40]
-
[41]
H. Thasarathan, J. Forsyth, T. Fel, M. Kowal, and K. G. Derpanis. Universal sparse autoencoders: Interpretable cross-model concept alignment.Proceedings of Machine Learning Research, 267, 2025
work page 2025
- [42]
-
[43]
A. H. Williams, E. Kunz, S. Kornblith, and S. Linderman. Generalized shape metrics on neural represen- tations. InAdvances in Neural Information Processing Systems, volume 34, pages 4738–4750. Curran Associates, 2021
work page 2021
-
[44]
N. Kriegeskorte, M. Mur, D. A. Ruff, R. Kiani, J. Bodurka, H. Esteky, K. Tanaka, and P. A. Bandettini. Matching categorical object representations in inferior temporal cortex of man and monkey.Neuron, 60: 1126–1141, 2008
work page 2008
-
[45]
B. Schölkopf, A. Smola, and K.-R. Müller. Kernel principal component analysis. InInternational Conference on Artificial Neural Networks, volume 1327, pages 583–588. Springer, 1997
work page 1997
-
[46]
D. D. Lee and H. S. Seung. Algorithms for non-negative matrix factorization.Advances in Neural Information Processing Systems, 13:556–562, 2000
work page 2000
-
[47]
Q. Shi, H. Sun, S. Lu, M. Hong, and M. Razaviyayn. Inexact block coordinate descent methods for symmetric nonnegative matrix factorization.IEEE Transactions on Signal Processing, 65:5995–6008, 2017
work page 2017
- [48]
-
[49]
A. R. Zamir, A. Sax, W. Shen, L. J. Guibas, J. Malik, and S. Savarese. Taskonomy: disentangling task transfer learning. InIEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3712–3722. IEEE, 2018
work page 2018
-
[50]
N. Mu, A. Kirillov, D. Wagner, and S. Xie. SLIP: Self-supervision meets language-image pre-training. In European Conference on Computer Vision, pages 529–544. Springer, 2022
work page 2022
-
[51]
I. Sucholutsky, L. Muttenthaler, A. Weller, A. Peng, A. Bobu, B. Kim, B. C. Love, C. J. Cueva, E. Grant, I. Groen, J. Achterberg, J. B. Tenenbaum, K. M. Collins, K. L. Hermann, K. Oktar, K. Greff, M. N. Hebart, N. Cloos, N. Kriegeskorte, N. Jacoby, Q. Zhang, R. Marjieh, R. Geirhos, S. Chen, S. Kornblith, S. Rane, T. Konkle, T. P. O’Connell, T. Unterthiner...
work page 2025
- [52]
-
[53]
O. Schoppe, N. S. Harper, B. D. B. Willmore, A. J. King, and J. W. H. Schnupp. Measuring the performance of neural models.Frontiers in Computational Neuroscience, 10:10, 2016. 13 A Model Overview While no finite benchmark can represent all conceivable vision models, our set of 162 models span major contemporary sources of variation in visual representatio...
work page 2016
-
[54]
vs the mean across 1,000 bootstrap resamples (20% of models, n= 32 ).(b)Leave-family-out stability: universality from the full set vs recomputed after excluding all models from the same architecture family. Leave-family-out stability.We also test whether the universality of dimensions in a given model depends on other models from the same architectural fa...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.