REVIEW 3 major objections 5 minor 45 references
Self-supervised vision transformers, especially DINO models, group objects in natural scenes in ways that track human reaction times, and that alignment is driven by how similar their patches are within the same object.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-14 21:33 UTC pith:TO5OSERQ
load-bearing objection Solid new human benchmark + object-centric AUC + Gram-distillation intervention; DINO transformers win on behavioral alignment, and the comparative evidence holds even if the MLP readout is imperfect. the 3 major comments →
Human-like Object Grouping in Self-supervised Vision Transformers
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Self-supervised vision models, above all DINO-trained transformers, capture object structure in a behaviorally human-like way: the strength of their patch-level object-centric structure (measured by ROC analysis of affinity maps) predicts how well a simple readout from those patches predicts human reaction times on same/different object judgments, and matching a supervised model’s Gram matrix of patch similarities to a DINO model’s improves both object-centricity and behavioral alignment.
What carries the argument
Object-centric AUC: for each trial an affinity map is built from cosine similarities of the central-dot patch to every other patch; ROC analysis of how well those similarities recover the ground-truth object mask yields an AUC that quantifies object-centric structure without object-level supervision. The Gram matrix of all pairwise patch similarities is the related global object whose alignment, via distillation, transfers that structure.
Load-bearing premise
That a two-layer MLP reading only the last-layer features at the two fixed dot locations is a fair, high-capacity probe of each model’s grouping knowledge, so differences in prediction accuracy can be blamed on the models rather than on the readout or the training trials.
What would settle it
If a model family with low object-centric AUC nevertheless matched or exceeded DINO models in noise-normalized Spearman correlation with human RTs under the same readout protocol, or if Gram-matrix distillation failed to raise both AUC and RT correlation for the supervised models, the claimed link would be undermined.
If this is right
- Training objectives that explicitly shape patch-similarity structure (Gram anchoring, self-distillation) should be preferred over pure classification when the goal is human-like object grouping.
- Object-centric AUC can serve as a cheap, label-free diagnostic of whether a new vision model is likely to align with human segmentation behavior.
- Convolutional architectures can also acquire strong object-centric structure when distilled from a DINO teacher, so architecture alone is not the bottleneck.
- A remaining gap to the human noise ceiling implies that further gains are still available by refining how similarity structure is shaped during pre-training.
Where Pith is reading between the lines
- The same Gram-alignment recipe could be tested as a post-hoc fix for any dense vision backbone whose object grouping currently lags humans, including models never trained with DINO.
- Because the metric needs only patch features and object masks, it can be applied to intermediate layers or attention heads to locate where object-centric structure first appears, guiding architecture search.
- If the two-dot RT benchmark were extended to multi-object clutter or partial occlusion, the same models would likely reveal where current object-centric representations still break.
- The finding that perfect object-centricity is not the goal suggests an optimal trade-off between object clustering and fine feature discrimination that future losses could target explicitly.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper scales a classical two-dot same/different object judgment paradigm to 255 naturalistic COCO scenes (1020 trials) and uses human reaction times as a behavioral benchmark of object grouping. A diverse set of public vision models is probed with a two-layer MLP readout on concatenated last-layer patch features at the two dot locations, predicting both grouping accuracy and trial-level RTs (noise-normalized Spearman). The authors introduce an object-centric AUC metric based on ROC analysis of patch affinity maps against COCO object masks, show that this metric strongly tracks behavioral alignment across models, and demonstrate that distilling the Gram matrix of DINOv3 into supervised ViT and ConvNeXt models improves grouping accuracy, object-centric AUC, and RT alignment. DINO-family transformers lead on all three metrics; training objective appears more important than architecture alone.
Significance. If the results hold, the work provides one of the cleaner links between emergent object structure in self-supervised vision transformers and human perceptual grouping under naturalistic conditions. Strengths include an external behavioral measure (RTs with same-object advantage and distance interaction), a label-free object-centric metric defined from masks rather than RTs, consistent cross-model ordering on three related metrics, and an interventional Gram-distillation experiment that improves all three on four supervised models—converging with DINOv3’s own Gram-anchoring design. The scaled two-dot benchmark and the object-centric AUC are reusable contributions for both computer vision and vision science.
major comments (3)
- [Section 4, Behavior Prediction] Section 4 (Modeling Experiments / Behavior Prediction): The central comparative claims rest on a two-layer MLP readout from concatenated last-layer patch features at the two fixed dots, trained either on ~30k synthetic COCO trials (accuracy) or 10-fold CV on the 1020 behavioral trials (RTs). The manuscript does not show that model ranking is robust to a simpler, lower-capacity probe (e.g., cosine similarity of the two patch tokens alone, or a linear classifier). Without that control, differences could partly reflect how well each representation fits this particular MLP rather than intrinsic grouping structure. A short ablation with at least one linear/cosine baseline would make the attribution to the models much firmer.
- [Section 4.3, Fig. 7] Section 4.3 (Gram Alignment) and Fig. 7: The intervention is load-bearing for the claim that Gram structure drives perceptual alignment, but the distillation recipe is underspecified—no loss equation, relative weight of Gram vs. ImageNet classification, which layers are matched, or training schedule. Reproducibility and interpretation of the large gains (especially on Transformers without large-scale pretraining) require these details, ideally with a brief ablation on Gram-loss weight.
- [Section 4.2, Fig. 6B] Fig. 6B and the associated claim that object-centric AUC “strongly predicts” behavioral alignment: with only ~9–10 models the scatter is informative but needs a reported correlation coefficient, confidence interval or permutation p-value, and ideally a leave-one-model-out check so that the relationship is not driven by the DINO cluster alone. This is the main quantitative bridge between the proposed metric and human behavior.
minor comments (5)
- [Section 3.1] Section 3.1: Report how many images were discarded in the manual inspection step after algorithmic placement, and whether any quantitative check was done that the retained set is not biased toward unusually clean object boundaries.
- [Table 1, Section 4] Table 1 and Fig. 4: DINOv3 ConvNeXt B is listed under “Distilled”; the text sometimes treats it as evidence that convolutional architectures can host object-centric structure. Clarify that its features come from a DINO teacher so it is not an independent architecture-only comparison.
- [Fig. 1, Fig. 5] Fig. 1 and Fig. 5: Affinity maps and thresholded examples are helpful; adding a brief note on whether features are L2-normalized before cosine similarity (standard for Gram/affinity) would remove ambiguity.
- [Conclusion / end matter] Data and code availability: The behavioral trial set (images, dot coordinates, mean RTs) and Gram-distillation code are not mentioned as released. Making them public would substantially increase the paper’s impact as a benchmark.
- [Throughout] Minor typos and formatting: “Toquantify” (Section 4.2), “weappliedthesamedotplacementalgorithm” (Section 4), and a few missing spaces in figure captions; also “Effect of Gram alignmen” is truncated in Fig. 7’s title.
Circularity Check
No load-bearing circularity; human RTs, COCO masks, and public checkpoints are independent of the models under test. The sole mild near-tautology is that Gram distillation improves object-centric AUC (a direct summary of affinities).
specific steps
-
other
[Section 4.3 Gram Alignment / Fig. 7B]
"Gram alignment consistently improves grouping accuracy across all four models... A similarly consistent improvement is observed in object-centric AUC (Fig. 7B), with all models showing substantially higher object-centricity after Gram alignment"
Object-centric AUC is defined by ROC analysis of thresholded affinity maps (cosine similarities of patch tokens), which are precisely the rows of the Gram matrix. Distilling the Gram of a high-AUC teacher (DINOv3) therefore raises the student’s AUC largely by construction whenever the distillation succeeds; the reported AUC gains are not an independent discovery of object-centric structure.
full rationale
The derivation chain is self-contained against external benchmarks. Human same/different RTs are new behavioral data collected under controlled conditions; model features come from independently trained public checkpoints (DINO family, MAE, DeiT3, ConvNeXt); the object-centric ROC-AUC is computed solely from patch affinities versus COCO ground-truth masks and never uses RT labels; the cross-model correlation (Fig. 6B) and the Gram-distillation intervention (Fig. 7) are therefore comparative and interventional rather than definitional. The MLP readout is a standard linear probe trained either on held-out synthetic COCO trials or via 10-fold CV; it does not fit parameters that are later re-presented as predictions of the same quantity. Self-citations exist but are not load-bearing uniqueness theorems. The only minor reduction is that object-centric AUC is itself a function of the affinity structure that the Gram matrix encodes, so successful Gram matching to a high-AUC teacher necessarily raises AUC; this is acknowledged as near-tautological for that intermediate metric alone and does not force the behavioral-alignment gains that constitute the central claim. Score remains low because the paper’s strongest evidence (ordering of models, AUC–behavior correlation, and positive intervention on held-out RTs) survives even if the AUC improvement is partly by construction.
Axiom & Free-Parameter Ledger
free parameters (4)
- MLP readout architecture and training hyperparameters
- Affinity thresholds for ROC / TPR-FPR curves
- Noise-ceiling split procedure (20 random half-splits)
- Dot-placement algorithm distance and overlap constraints
axioms (4)
- domain assumption Reaction time on correct same/different object judgments indexes the difficulty of perceptual grouping/segmentation in natural scenes.
- domain assumption Cosine similarity of last-layer patch tokens is an appropriate measure of feature affinity for object-centricity.
- ad hoc to paper A simple linear-ish readout from concatenated patch features at two locations is sufficient to compare models’ grouping representations.
- domain assumption COCO instance masks provide ground-truth object boundaries adequate for defining within- vs. between-object patches.
invented entities (2)
-
Object-centric AUC metric (ROC of patch affinity vs. object mask)
independent evidence
-
Scaled two-dot naturalistic behavioral benchmark (~1020 trials)
independent evidence
read the original abstract
Vision foundation models trained with self-supervised objectives achieve strong performance across diverse tasks and exhibit emergent object segmentation properties. However, their alignment with human object perception remains poorly understood. Here, we introduce a behavioral benchmark in which participants make same/different object judgments for dot pairs on naturalistic scenes, scaling up a classical psychophysics paradigm to over 1000 trials. We test a diverse set of vision models using a simple readout from their representations to predict subjects' reaction times. We observe a steady improvement across model generations, with both architecture and training objective contributing to alignment, and transformer-based models trained with the DINO self-supervised objective showing the strongest performance. To investigate the source of this improvement, we propose a novel metric to quantify the object-centric component of representations by measuring patch similarity within and between objects. Across models, stronger object-centric structure predicts human segmentation behavior more accurately. We further show that matching the Gram matrix of supervised transformer models, capturing similarity structure across image patches, with that of a self-supervised model through distillation improves their alignment with human behavior, converging with the prior finding that Gram anchoring improves DINOv3's feature quality. Together, these results demonstrate that self-supervised vision models capture object structure in a behaviorally human-like manner, and that Gram matrix structure plays a role in driving perceptual alignment.
Figures
Reference graph
Works this paper leans on
-
[1]
Journal of Vision23(5), 16–16 (2023) 1
Adeli, H., Ahn, S., Zelinsky, G.J.: A brain-inspired object-based attention network for multiobject recognition and visual reasoning. Journal of Vision23(5), 16–16 (2023) 1
2023
-
[2]
arXiv preprint arXiv:2505.17329 (2025) 3
Adeli, H., Sun, M., Kriegeskorte, N.: Transformer brain encoders explain human high-level visual responses. arXiv preprint arXiv:2505.17329 (2025) 3
arXiv 2025
-
[3]
PLOS Computational Biology20(6), e1012159 (2024) 1
Ahn, S., Adeli, H., Zelinsky, G.J.: The attentive reconstruction of objects facilitates robust object recognition. PLOS Computational Biology20(6), e1012159 (2024) 1
2024
-
[4]
arXiv preprint arXiv:2112.05814 (2021) 3
Amir, S., Gandelsman, Y., Bagon, S., Dekel, T.: Deep vit features as dense visual descriptors. arXiv preprint arXiv:2112.05814 (2021) 3
Pith/arXiv arXiv 2021
-
[5]
bioRxiv pp
Biswas, T.K., Vacher, J., Molholm, S., Mamassian, P., Coen-Cagli, R.: Natural scene segmentation dynamics reveal iterative bayesian inference. bioRxiv pp. 2026–01 (2026) 1
2026
-
[6]
In: Proceedings of the IEEE/CVF international conference on computer vision
Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self- supervised vision transformers. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 9650–9660 (2021) 1, 3, 5
2021
-
[7]
arXiv preprint arXiv:2505.21501 (2025) 3
Chen, Y., Yan, Z., Zhou, C., Dai, B., Luo, A.F.: Vision transformers with self-distilled registers. arXiv preprint arXiv:2505.21501 (2025) 3
arXiv 2025
-
[8]
Coen-Cagli,R.,Mamassian,P.:Arewereadytotackleperceptualsegmentationofnaturalscenes?VisionResearch 240, 108749 (2026) 2
2026
-
[9]
arXiv preprint arXiv:2309.16588 (2023) 5
Darcet, T., Oquab, M., Mairal, J., Bojanowski, P.: Vision transformers need registers. arXiv preprint arXiv:2309.16588 (2023) 5
Pith/arXiv arXiv 2023
-
[10]
bioRxiv pp
Dedieu, A., Rikhye, R.V., Lázaro-Gredilla, M., George, D.: Learning attention-controllable border-ownership for objectness inference and binding. bioRxiv pp. 2020–12 (2021) 1
2020
-
[11]
arXiv preprint arXiv:2507.00493 (2025) 3
Doshi,F.R.,Fel,T.,Konkle,T.,Alvarez,G.:Visualanagramsrevealhiddendifferencesinholisticshapeprocessing across vision models. arXiv preprint arXiv:2507.00493 (2025) 3
arXiv 2025
-
[12]
arXiv preprint arXiv:2010.11929 (2020) 3, 5
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020) 3, 5
Pith/arXiv arXiv 2010
-
[13]
Journal of Experimental Psychology: General123(2), 161 (1994) 5
Egly, R., Driver, J., Rafal, R.D.: Shifting visual attention between objects and locations: evidence from normal and parietal lesion subjects. Journal of Experimental Psychology: General123(2), 161 (1994) 5
1994
-
[14]
associ- ation field
Field, D.J., Hayes, A., Hess, R.F.: Contour integration by the human visual system: evidence for a local “associ- ation field”. Vision research33(2), 173–193 (1993) 1, 10
1993
-
[15]
Science358(6368), eaag2612 (2017) 1 12
George, D., Lehrach, W., Kansky, K., Lázaro-Gredilla, M., Laan, C., Marthi, B., Lou, X., Meng, Z., Liu, Y., Wang, H., et al.: A generative vision model that trains with high data efficiency and breaks text-based captchas. Science358(6368), eaag2612 (2017) 1 12
2017
-
[16]
Nature reviews neuroscience14(5), 350–363 (2013) 1
Gilbert, C.D., Li, W.: Top-down influences on visual processing. Nature reviews neuroscience14(5), 350–363 (2013) 1
2013
-
[17]
arXiv preprint arXiv:2012.05208 (2020) 1
Greff, K., van Steenkiste, S., Schmidhuber, J.: On the binding problem in artificial neural networks. arXiv preprint arXiv:2012.05208 (2020) 1
Pith/arXiv arXiv 2012
-
[18]
Advances in Neural Information Processing Systems37, 53732–53763 (2024) 1
Han, S., Wang, Z., Zhang, M.: Flow snapshot neurons in action: Deep neural networks generalize to biological motion perception. Advances in Neural Information Processing Systems37, 53732–53763 (2024) 1
2024
-
[19]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., Girshick, R.: Masked autoencoders are scalable vision learners. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 16000–16009 (2022) 1, 3, 6
2022
-
[20]
arXiv preprint arXiv:2510.21142 (2025) 3
Hwang, E., Adeli, H., Guo, W., Luo, A., Kriegeskorte, N.: In silico mapping of visual categorical selectivity across the whole brain. arXiv preprint arXiv:2510.21142 (2025) 3
arXiv 2025
-
[21]
Elife5, e14320 (2016) 1
Jeurissen, D., Self, M.W., Roelfsema, P.R.: Serial grouping of 2d-image regions with object-based attention in humans. Elife5, e14320 (2016) 1
2016
-
[22]
arXiv preprint arXiv:1906.01558 (2019) 1, 3
Kim, J., Linsley, D., Thakkar, K., Serre, T.: Disentangling neural mechanisms for perceptual grouping. arXiv preprint arXiv:1906.01558 (2019) 1, 3
Pith/arXiv arXiv 1906
-
[23]
Psychological Science23(12), 1482–1489 (2012) 3
Korjoukov, I., Jeurissen, D., Kloosterman, N.A., Verhoeven, J.E., Scholte, H.S., Roelfsema, P.R.: The time course of perceptual grouping in natural scenes. Psychological Science23(12), 1482–1489 (2012) 3
2012
-
[24]
arXiv preprint arXiv:2209.15172 (2022) 3
Lee, H.H., Chang, A.X.: Understanding pure clip guidance for voxel grid nerf models. arXiv preprint arXiv:2209.15172 (2022) 3
Pith/arXiv arXiv 2022
-
[25]
arXiv preprint arXiv:2506.00718 (2025) 3
Li, T., Wen, Z., Song, L., Liu, J., Jing, Z., Lee, T.S.: From local cues to global percepts: Emergent gestalt organization in self-supervised vision models. arXiv preprint arXiv:2506.00718 (2025) 3
Pith/arXiv arXiv 2025
-
[26]
Li, Y., Salehi, S., Ungar, L., Kording, K.P.: Does object binding naturally emerge in large pretrained vision transformers? arXiv preprint arXiv:2510.24709 (2025) 3
arXiv 2025
-
[27]
In: Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13
Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13. pp. 740–755. Springer (2014) 4
2014
-
[28]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Liu, Z., Mao, H., Wu, C.Y., Feichtenhofer, C., Darrell, T., Xie, S.: A convnet for the 2020s. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 11976–11986 (2022) 6
2022
-
[29]
arXiv preprint arXiv:2504.05253 (2025) 2
Lonnqvist, B., Scialom, E., Gokce, A., Merchant, Z., Herzog, M.H., Schrimpf, M.: Contour integration underlies human-like vision. arXiv preprint arXiv:2504.05253 (2025) 2
Pith/arXiv arXiv 2025
-
[30]
arXiv preprint arXiv:2303.01542 (2023) 3
Mehrani, P., Tsotsos, J.K.: Self-attention in vision transformers performs perceptual grouping, not attention. arXiv preprint arXiv:2303.01542 (2023) 3
Pith/arXiv arXiv 2023
-
[31]
Oquab, M., Darcet, T., Moutakanni, T., Vo, H.V., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., Howes, R., Huang, P.Y., Xu, H., Sharma, V., Li, S.W., Galuba, W., Rabbat, M., Assran, M., Ballas, N., Synnaeve, G., Misra, I., Jegou, H., Mairal, J., Labatut, P., Joulin, A., Bojanowski, P.: Dinov2: Learning robust visual feat...
2023
-
[32]
PloS one20(12), e0337087 (2025) 1
Papale, P., Williford, J.R., Balk, S., Roelfsema, P.R.: Modulatory feedback determines attentional object seg- mentation in a model of the ventral stream. PloS one20(12), e0337087 (2025) 1
2025
-
[33]
Nature Human Behaviour pp
Peters, B., Kriegeskorte, N.: Capturing the objects of vision with neural networks. Nature Human Behaviour pp. 1–18 (2021) 1
2021
-
[34]
Neuron111(7), 1003–1019 (2023) 1, 5, 10
Roelfsema, P.R.: Solving the binding problem: Assemblies form when neurons enhance their firing rate—they don’t need to oscillate or synchronize. Neuron111(7), 1003–1019 (2023) 1, 5, 10
2023
-
[35]
arXiv preprint arXiv:2109.14279 (2021) 3
Siméoni, O., Puy, G., Vo, H.V., Roburin, S., Gidaris, S., Bursuc, A., Pérez, P., Marlet, R., Ponce, J.: Localizing objects with self-supervised transformers and no labels. arXiv preprint arXiv:2109.14279 (2021) 3
Pith/arXiv arXiv 2021
-
[36]
arXiv preprint arXiv:2508.10104 (2025) 1, 2, 3, 5, 11
Siméoni, O., Vo, H.V., Seitzer, M., Baldassarre, F., Oquab, M., Jose, C., Khalidov, V., Szafraniec, M., Yi, S., Ramamonjisoa, M., et al.: Dinov3. arXiv preprint arXiv:2508.10104 (2025) 1, 2, 3, 5, 11
Pith/arXiv arXiv 2025
-
[37]
arXiv preprint arXiv:2106.10270 (2021) 6
Steiner, A., Kolesnikov, A., Zhai, X., Wightman, R., Uszkoreit, J., Beyer, L.: How to train your vit? data, augmentation, and regularization in vision transformers. arXiv preprint arXiv:2106.10270 (2021) 6
Pith/arXiv arXiv 2021
-
[38]
bioRxiv pp
Toosi, T., Miller, K.D.: Generative inference unifies feedback processing for learning and perception in natural and artificial vision. bioRxiv pp. 2025–10 (2025) 1
2025
-
[39]
In: European conference on computer vision
Touvron, H., Cord, M., Jégou, H.: Deit iii: Revenge of the vit. In: European conference on computer vision. pp. 516–533. Springer (2022) 6
2022
-
[40]
Current opinion in neurobiology6(2), 171–178 (1996) 1 13
Treisman, A.: The binding problem. Current opinion in neurobiology6(2), 171–178 (1996) 1 13
1996
-
[41]
Brain and Mind 1(3), 353–384 (2000) 1, 3
Vecera, S.P.: Toward a biased competition account of object-based segregation and attention. Brain and Mind 1(3), 353–384 (2000) 1, 3
2000
-
[42]
perceptual grouping and figure–ground organization
Wagemans, J., Elder, J.H., Kubovy, M., Palmer, S.E., Peterson, M.A., Singh, M., von der Heydt, R.: A century of gestalt psychology in visual perception: I. perceptual grouping and figure–ground organization. Psychological bulletin138(6), 1172 (2012) 1
2012
-
[43]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Wang, Y., Shen, X., Hu, S.X., Yuan, Y., Crowley, J.L., Vaufreydaz, D.: Self-supervised transformers for unsuper- vised object discovery using normalized cut. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 14543–14553 (2022) 3
2022
-
[44]
Wang, Z., Han, S., Shou, M.Z., Zhang, M.: Unsupervised prior learning: Discovering categorical pose priors from videos (2024) 1
2024
-
[45]
Advances in Neural Information Processing Systems36, 9388–9408 (2023) 1 14
Wang, Z., Shou, M.Z., Zhang, M.: Object-centric learning with cyclic walks between parts and whole. Advances in Neural Information Processing Systems36, 9388–9408 (2023) 1 14
2023
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.