Pith. sign in

REVIEW 3 major objections 5 minor 78 references

MoSiC: Optimal-Transport Motion Trajectory for Dense Self-Supervised Learning

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper proposes that unlabeled video motion, channeled through point-track cluster propagation, can sharpen the dense features of a frozen-initialized image backbone such as DINOv2.

desk verdict The loss in eq. (10) does not supervise later frames, so the paper's central claim is unsupported; the empirical study is solid but needs a fix. read the letter →

arxiv 2506.08694 v2 pith:V73PTUZN submitted 2025-06-10 cs.CV

Mohammadreza Salehi , Shashanka Venkataramanan , Ioana Simion , Efstratios Gavves , Cees G. M. Snoek , Yuki M Asano This is my paper
classification cs.CV
keywords self-supervisedlearningdenserepresentationpointtrackingoptimaltransportclusteringvideoobjectsegmentationDINOv2temporalconsistencysemantic
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces MoSiC, a self-supervised method that turns point trajectories into a temporal supervisory signal for dense visual representations. It clusters first-frame features with an entropy-regularized optimal transport step, propagates those cluster assignments along tracked points across frames, and trains the student's first-frame cluster scores against the propagated teacher targets. The claim is that this motion-guided clustering improves dense representations without any labels, yielding 1 to 6 percent gains over DINOv2 across four benchmarks and 8.7 to 9.4 percent gains over TimeT on video segmentation. If correct, unlabeled video alone can improve downstream semantic segmentation, in-context scene understanding, and video object segmentation.

What carries the argument

The load-bearing mechanism is trajectory-based cluster propagation. An off-the-shelf point tracker samples a grid of points in the first frame and produces long-range trajectories (eq. 1); teacher and student features are clustered into K prototypes via entropy-regularized optimal transport using Sinkhorn-Knopp; the teacher's first-frame hard assignments are copied along each trajectory (eq. 8); and the student is trained to reproduce those propagated assignments at the first frame for points that remain visible (eq. 10). The visibility flag restricts the loss to points that stay visible, which the ablations show is important for avoiding false similarity enforcement.

What would settle it

Measure, on a held-out video set, whether a propagated first-frame cluster assignment agrees with a fresh cluster assignment computed on the later frame's own features for the same track; if agreement is at chance for a substantial fraction of visible tracks, the cross-entropy signal is not carrying semantic identity.

Watch

Extended reading notes

Core claim

The central claim is that motion can act as an implicit dense label: points that move together should keep the same cluster identity over time. Concretely, MoSiC computes Sinkhorn-Knopp optimal-transport cluster assignments for the first frame of a clip, propagates each tracked point's teacher assignment along its trajectory, and applies a cross-entropy loss (eq. 10) between the student's first-frame cluster scores and the propagated teacher one-hot targets, masked to visible points. The paper asserts that this mechanism mitigates occlusions and long-range tracking drift, and that training on YouTube-VOS consistently improves the dense features of a DINOv2-initialized ViT across four image and video benchmarks.

Load-bearing premise

The method assumes the point tracker keeps each point attached to the same physical object across the clip, so a propagated cluster label is semantically correct; when a track drifts, the loss trains the student to merge or split features incorrectly.

Editorial extensions

If this is right

  • Temporal motion alone, without any dense annotations, can improve the dense features of a strong image-pretrained backbone.
  • Propagating cluster identities along tracks gives cleaner unsupervised video object segmentation than mask propagation, with 8.7 to 9.4 mIoU gains over TimeT.
  • In-context scene understanding improves most in low-data regimes, suggesting motion pretraining helps feature transfer.
  • The recipe transfers to other backbones, including vision-language encoders, and scales with model size.
  • End-to-end fine-tuning on videos alone can improve object detection and semantic segmentation over the image-only backbone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to verify track-object consistency by cycle consistency or mask agreement before propagating labels; the paper's failure cases suggest filtering bad tracks would add further gains.
  • If point trackers keep improving, tracker quality rather than clustering becomes the bottleneck; the paper's tracker ablation already points in this direction.
  • The same propagation objective could be applied to multi-object labels over longer clips, potentially replacing mask propagation in label-efficient video segmentation pipelines.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes MoSiC, a dense self-supervised video pretraining method. It extracts point trajectories with CoTracker-v3, computes Sinkhorn-Knopp cluster assignments on the first frame for both teacher and student networks, then claims to propagate the teacher's first-frame cluster assignments along the trajectories and trains the student with a cross-entropy loss. The method is initialized from DINOv2, trained on YouTube-VOS, and evaluated on unsupervised video object segmentation, in-context scene understanding, frozen clustering, and linear segmentation, reporting gains over DINOv2 and TimeT.

Significance. If the stated mechanism were actually implemented, the idea of using long-range point tracks to provide dense temporal supervision for a strong image-pretrained backbone is timely and potentially valuable. The paper is strongest in its controlled comparisons against DINOv2 with identical architecture and initialization, where it reports consistent albeit modest gains (about 1 to 4 points on linear segmentation and clustering, up to 6 points in low-data in-context retrieval). The authors provide code, extensive ablations, and a candid discussion of tracker failure cases. However, as written, the central loss does not use any features from frames after t0, so the claimed temporal propagation mechanism is not realized in the stated equations. This issue must be resolved before the paper's significance can be fairly assessed.

major comments (3)
  1. [Section 3.4, Eqs. (8)-(10)] The loss in Eq. (10) does not implement temporal cluster propagation. The student score S^{stu,k,i}_{t0} in Eq. (9) depends only on Z^{stu,i}_{t0}, and Eq. (8) sets Q^{teach,i}_t = Q^{teach,i}_{t0} for every t. Consequently, Eq. (10) factors as -(sum_t v_{t,i}) * sum_k delta(Q^{teach,i}_{t0}=k) * log S^{stu,k,i}_{t0}, which is a first-frame cross-entropy clustering loss weighted by the total visible-track length. The gradient with respect to student features at frames t>0 is identically zero, and the teacher's later-frame features are never used. The central claim that the method 'propagates cluster assignments along motion trajectories' to enforce spatiotemporal consistency is therefore not supported by the stated objective. The authors must either correct the equations to use per-frame student scores and per-frame teacher assignments, or revise the claimed mechanism to match the actual loss.
  2. [Section 4.2, Table 2] The headline comparison against TimeT is confounded. MoSiC-S14 uses a ViT-S/14 initialized from DINOv2, while TimeT uses a ViT-S/16 initialized from DINO. The footnote in Table 2 acknowledges the architecture difference but does not control for it, so the reported '8.7% and 9.4% mIoU' gains over TimeT partly reflect the stronger backbone and initialization rather than the proposed method. The controlled comparisons against DINOv2-S/14 (e.g., +4.3 and +3.7 on average) are much more modest and should be the primary basis for the paper's claims, or an ablation with a common backbone and initialization should be added.
  3. [Section 4.7, Ablations] The ablations do not isolate the temporal component of the method. All reported ablations vary parameters such as mask ratio, grid size, number of prototypes, clip length, and frame count, but there is no baseline that trains the same first-frame clustering objective without the trajectory-derived visibility weighting. Given that Eq. (10) reduces to a first-frame loss with visibility weights, the observed gains over DINOv2 could come from the weighting or from the extra training data rather than from any spatiotemporal consistency enforced along tracks. A single-frame training baseline with identical loss (e.g., setting all v_{t,i}=1 or using one frame per clip) is necessary to support the claim that motion trajectories provide the supervisory signal.
minor comments (5)
  1. [Figure 3 caption] The caption appears to swap the equation references: it attributes Q^{stu}_{t0} to Eq. (7) and Q^{teach}_{t0} to Eq. (6), while in the text Eq. (6) defines Q^{stu}_{t0} and Eq. (7) defines Q^{teach}_{t0}.
  2. [Section 3.2] There is a typo: 'epsilon is the regularization coefficient than controls' should be 'that controls'.
  3. [Section 3.3, Eqs. (6)-(9)] The notation in Eqs. (6) and (7) is dimensionally unclear: M^{stu*} is introduced as an assignment matrix in R^{n_s x K}, but it is multiplied by Z^{stu}_{t0} in R^{n_s x d}, and in Eq. (9) it is treated as if it were the prototype matrix. Please clarify the exact matrix products and the roles of M and P.
  4. [Section 4.7, Table 6h] The phrase 'time step per frame fixed at 3.2/12 seconds' is ambiguous; it should state the actual temporal stride in seconds between consecutive frames.
  5. [Section 7.2, Figure 8] The failure-case discussion is not reconciled with Eq. (10). As written, a drifted track that remains visible only changes the scalar visibility weight; it does not propagate a wrong cluster label to a later-frame destination feature, because the teacher label is frozen at t0. The authors should clarify how the illustrated tracker failure affects the actual loss under their formulation.

Circularity Check

1 steps flagged · score 6.0 of 10

Eq. (10) reduces the claimed temporal-cluster-propagation loss to a visibility-weighted first-frame cross-entropy, so the central motion mechanism is by construction the initial-frame assignment.

  1. self definitional [Section 3.3–3.4, Eqs. (8)–(10)]
    "Q^{teach,i}_t = Q^{teach,i}_{t0}, \forall (x_{i,t}, y_{i,t}) = Traj_{t,i} (8) ... L_{clust}(i) = -\sum_{t=1}^T \sum_{k=1}^K v_{t,i} \cdot \delta(Q^{teach,i}_t = k) \cdot \log(S^{stu,k,i}_{t0}) (10)"

    Substituting Eq. (8) into Eq. (10) gives L_clust(i) = -(\sum_t v_{t,i}) \sum_k \delta(Q^{teach,i}_{t0}=k) \log S^{stu,k,i}_{t0}. The student score in Eq. (9) is computed from Z^{stu,i}_{t0} only, and Eq. (8) defines the propagated teacher label to be identical to the first-frame label. Trajectories beyond t0 therefore enter only through the scalar visibility count; no student or teacher feature at any t>0 appears. The loss is exactly a visibility-weighted first-frame cross-entropy clustering objective.

full rationale

The paper is otherwise self-contained: the student–teacher Sinkhorn clustering is the standard DINO-family bootstrap, CoTracker-v3 and DINOv2 are external off-the-shelf components, and the headline evaluations are against external labels (Pascal VOC, ADE20K, COCO, DAVIS, YouTube-VOS) using Hungarian matching. Author self-citations such as TimeT, Leopart, NeCo, and Sigma are baselines or prior architectural components rather than load-bearing justifications for the central claim. The reader's note about track drift is a correctness risk, not a circularity. The one decisive circular step is Eq. (10): because Eq. (8) defines the propagated label as equal to the t0 label and Eq. (9) uses only t0 student features, the 'temporal propagation' loss is algebraically a weighted first-frame clustering loss, with no gradient to features at t>0. The abstract's statement that the method 'propagates cluster assignments along tracked points, enforcing feature consistency across views' is therefore not realized by the stated loss; this is a self-definitional reduction of the claimed mechanism. Hyperparameters selected on the same Pascal VOC/ADE20K validation sets used for reported gains are a model-selection concern but do not constitute a derivation-level circularity. Score 6 reflects that one core 'prediction'—temporal coherence from motion—reduces by construction, while the empirical evaluations remain externally grounded.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The method rests on standard SSL machinery plus a domain assumption about motion semantics. No invented entities. Free parameters are the usual hyperparameters, several of which (masking ratio, grid size, prototype count, clip length, crop scale) were tuned on the very benchmarks used for the headline claims.

free parameters (6)
  • Sinkhorn entropy regularization epsilon = not reported
    Controls assignment smoothness in eq. (4)-(5); the value is never stated, making exact reproduction impossible.
  • Masking ratio m = 10%
    Ablated in Table 6a; 20% gives similar scores but 10% was chosen for 'training stability'.
  • Tracker grid size = 16 x 16 points
    Ablated in Table 6c; 8x8 and 32x32 both give worse results.
  • Number of prototypes K = 100
    Ablated in Table 6e; 200 gives similar ADE20K scores.
  • Crop scale interval = [0.4, 1]
    Ablated in Table 6d; chosen as the best of four intervals.
  • Clip length and frame count = 3.2 s, 12 frames
    Ablated in Tables 6g and 6h; gains grow with clip length and frame count.
assumptions (4)
  • domain assumption 'What moves together belongs together' (Gestalt grouping)
    Section 1. The entire supervisory signal assumes co-moving points share a semantic identity, which fails for articulated objects or for distinct parts that move in sync.
  • domain assumption EMA teacher features provide stable clustering targets
    Standard DINO-family assumption; the teacher's assignments are trusted as ground truth in eq. (10) without verification.
  • standard math Sinkhorn-Knopp converges to meaningful assignments for dense features
    Routine usage as in SwAV, DINO, and SIGMA; uniform marginal constraints force balanced clusters that may not match object sizes.
  • domain assumption DINOv2 initialization is preserved and improved by video post-training
    Section 4.1. The method's gains depend on the strong image pretraining; Table 7 shows classification drops by 0.8 percent.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MoSiC: Optimal-Transport Motion Trajectory for Dense Self-Supervised Learning." pith.science (2026). https://pith.science/paper/V73PTUZN

@misc{pith2026250608694,
  author       = {Pith},
  title        = {Pith review of: MoSiC: Optimal-Transport Motion Trajectory for Dense Self-Supervised Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V73PTUZN}},
  note         = {Machine review of arXiv:2506.08694}
}
read the original abstract

Dense self-supervised learning has shown great promise for learning pixel- and patch-level representations, but extending it to videos remains challenging due to the complexity of motion dynamics. Existing approaches struggle as they rely on static augmentations that fail under object deformations, occlusions, and camera movement, leading to inconsistent feature learning over time. We propose a motion-guided self-supervised learning framework that clusters dense point tracks to learn spatiotemporally consistent representations. By leveraging an off-the-shelf point tracker, we extract long-range motion trajectories and optimize feature clustering through a momentum-encoder-based optimal transport mechanism. To ensure temporal coherence, we propagate cluster assignments along tracked points, enforcing feature consistency across views despite viewpoint changes. Integrating motion as an implicit supervisory signal, our method learns representations that generalize across frames, improving robustness in dynamic scenes and challenging occlusion scenarios. By initializing from strong image-pretrained models and leveraging video data for training, we improve state-of-the-art by 1% to 6% on six image and video datasets and four evaluation benchmarks. The implementation is publicly available at our GitHub repository: https://github.com/SMSD75/MoSiC/tree/main

Figures

Figures reproduced from arXiv: 2506.08694 by the authors.

Figure 1
Figure 1. We introduce MoSiC, a self-supervised framework that uses optimal transport (OT) to enforce spatio-temporal consistency in dense visual representations. By tracking points across frames and clustering their motion trajectories, MoSiC encourages features of points that move together to be temporally coherent. videos proves ineffective [4, 57]. In images, correspondences are often implicitly encoded by data augmentati… view at source ↗
Figure 2
Figure 2. DINOv2 vs. MoSiC in-context scene understanding qualitative comparison on Pascal VOC. MoSiC trains DINOv2’s dense representations on unlabeled videos and improves them, resulting in more precise segmentation boundaries and better object identification. capture long-range motion trajectories across frames. These tracked points are then clustered using a momentum-encoder￾based optimal transport mechanism, ensuring fea… view at source ↗
Figure 3
Figure 3. MoSiC: Motion-based Sinkhorn Clustering for dense self-supervised pretraining. A video clip is patchified, with masked patches passed through the student network to obtain Z stu ∈ R ns×d×T , while the teacher network processes the patches from raw clip to produce Z teach ∈ R n×d×T . Features are clustered using Sinkhorn-Knopp [15] into prototypes P. The cluster assignments Q teach t0 (eq. (6)) and Q stu t0 (eq. (7))… view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Previous methods such as TimeT lose precise object [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: TimeT vs. MoSiC. As the figure shows, MoSiC consistently improves both DINO and DINOv2 backbones, while TimeT only improves DINO [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: Hummingbird qualitative results for MoSiC on Pascal VOC. The first group of rows displays the images, the second shows the corresponding per-image masks, and the third overlays the masks on the images, aligned by their semantic IDs. As shown, although MoSiC is finetune…
Figure 7
Figure 7. Figure 7: MoSiC overclustering visualizations on Pascal for K = 50. MoSiC not only localizes objects precisely but also identify them with different cluster ids demonstrated by different colors. For instance, classes such as birds, motorcycles, dogs, cats, and cars are clearly i…
Figure 8
Figure 8. Figure 8: Incorrect trajectories. The point tracks that start from the beak of the bird and eventually track the bowl. This happens due to the ambigious motion of the bird’s head [PITH_FULL_IMAGE:figures/full_fig_p021_8.png]
Figure 9
Figure 9. Figure 9: The distribution of classes in YouTube-VOS. Some of the [PITH_FULL_IMAGE:figures/full_fig_p021_9.png]
Figure 10
Figure 10. Figure 10: Pascal VOC visualizations. The first group of rows displays the images, the second shows the corresponding per-image masks, and the third overlays the masks on the images, aligned by their semantic IDs. These images and their ground truth segmentation maps are used fo…
Figure 11
Figure 11. Figure 11: DAVIS visualizations. The first group of rows displays the images, the second shows the corresponding per-image masks, and the third overlays the masks on the images, aligned by their semantic IDs. These images and their ground truth segmentation maps are used for our…
Figure 12
Figure 12. Figure 12: YouTube VOS visualizations. The first group of rows displays the images, the second shows the corresponding per-image masks, and the third overlays the masks on the images, aligned by their semantic IDs. These images and their ground truth segmentation maps are used f…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

78 extracted references · 69 canonical work pages

  1. [1]

    Learning to see by moving

    Pulkit Agrawal, Joao Carreira, and Jitendra Malik. Learning to see by moving. In ICCV, 2015. 2

    work page 2015

  2. [2]

    Dense unsupervised learning for video segmentation

    Nikita Araslanov, Simone Schaub-Meyer, and Stefan Roth. Dense unsupervised learning for video segmentation. NeurIPS, 2021. 2

    work page 2021

  3. [3]

    Self-labelling via simultaneous clustering and repre- sentation learning

    Yuki Markus Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultaneous clustering and repre- sentation learning. In ICLR, 2020. 4

    work page 2020

  4. [4]

    Self-supervised learning from images with a joint-embedding predictive architecture

    Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bo- janowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embedding predictive architecture. In CVPR, 2023. 1

    work page 2023

  5. [5]

    Towards in-context scene understanding

    Ivana Balazevic, David Steiner, Nikhil Parthasarathy, Relja Arandjelovi´c, and Olivier Henaff. Towards in-context scene understanding. NeurIPS, 2023. 2, 5, 6, 8, 15, 16

    work page 2023

  6. [6]

    Object discovery from motion- guided tokens

    Zhipeng Bao, Pavel Tokmakov, Yu-Xiong Wang, Adrien Gaidon, and Martial Hebert. Object discovery from motion- guided tokens. In ICCV, 2023. 2

    work page 2023

  7. [7]

    Revisiting feature prediction for learning visual representations from video

    Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mahmoud Assran, and Nico- las Ballas. Revisiting feature prediction for learning visual representations from video. TMLR, 2024. 1

    work page 2024

  8. [8]

    Coco- stuff: Thing and stuff classes in context

    Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco- stuff: Thing and stuff classes in context. In CVPR, 2018. 5, 7, 13, 14, 15, 17, 18

    work page 2018

Show all 78 references
  1. [9]

    Unsupervised learning of visual features by contrasting cluster assignments.NeurIPS, 33:9912–9924, 2020

    Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments.NeurIPS, 33:9912–9924, 2020. 4

    2020

  2. [10]

    Emerg- ing properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerg- ing properties in self-supervised vision transformers. InICCV,

  3. [11]

    Scaling 4d representations

    João Carreira, Dilara Gokay, Michael King, Chuhan Zhang, Ignacio Rocco, Aravindh Mahendran, Thomas Albert Keck, Joseph Heyward, Skanda Koppula, Etienne Pot, et al. Scaling 4d representations. arXiv preprint arXiv:2412.15212, 2024. 2

    2024 arXiv

  4. [12]

    Learning from one continuous video stream

    João Carreira, Michael King, Viorica Patraucean, Dilara Gokay, Catalin Ionescu, Yi Yang, Daniel Zoran, Joseph Hey- ward, Carl Doersch, Yusuf Aytar, et al. Learning from one continuous video stream. In CVPR, 2024. 2

    2024

  5. [13]

    Learn- ing to estimate pose by watching videos

    Prabuddha Chakraborty and Vinay P Namboodiri. Learn- ing to estimate pose by watching videos. arXiv preprint arXiv:1704.04081, 2017. 2

    2017 arXiv

  6. [14]

    A simple framework for contrastive learning of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geof- frey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020. 2

    2020

  7. [15]

    Sinkhorn distances: Lightspeed computation of optimal transport

    Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. NeurIPS, 2013. 3, 4

    2013

  8. [16]

    Vision transformers need registers

    Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In ICLR,

  9. [17]

    An image is worth 16x16 words: Trans- formers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale. In ICLR, 2021. 3, 14

    2021

  10. [18]

    Everingham, L

    M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results. http://www.pascal- network.org/challenges/VOC/voc2012/workshop/index.html,

    2012

  11. [19]

    Watching the world go by: Representation learning from unlabeled videos

    Daniel Gordon, Kiana Ehsani, Dieter Fox, and Ali Farhadi. Watching the world go by: Representation learning from unlabeled videos. arXiv preprint arXiv:2003.07990, 2020. 2

    2003 arXiv

  12. [20]

    Bootstrap your own latent-a new approach to self-supervised learning

    Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doer- sch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Ghesh- laghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. NeurIPS, 2020. 14

    2020

  13. [21]

    Accelerating large- scale inference with anisotropic vector quantization

    Ruiqi Guo, Philip Sun, Erik Lindgren, Quan Geng, David Simcha, Felix Chern, and Sanjiv Kumar. Accelerating large- scale inference with anisotropic vector quantization. In ICML,

  14. [22]

    Stego: Unsupervised se- mantic segmentation by distilling feature correspondences

    Mark Hamilton, Zhoutong Zhang, Bharath Hariharan, Noah Snavely, and William T Freeman. Stego: Unsupervised se- mantic segmentation by distilling feature correspondences. In ICLR, 2022. 2, 6, 16

    2022

  15. [23]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022. 6, 7, 15, 16

    2022

  16. [24]

    Effi- cient visual pretraining with contrastive detection

    Olivier J Hénaff, Skanda Koppula, Jean-Baptiste Alayrac, Aaron Van den Oord, Oriol Vinyals, and Joao Carreira. Effi- cient visual pretraining with contrastive detection. In ICCV,

  17. [25]

    Gaussian error linear units (gelus)

    Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016. 14

    2016 arXiv

  18. [26]

    Learning image representations tied to ego-motion

    Dinesh Jayaraman and Kristen Grauman. Learning image representations tied to ego-motion. In ICCV, 2015. 2

    2015

  19. [27]

    Invariant information clustering for unsupervised image classification and segmentation

    Xu Ji, Joao F Henriques, and Andrea Vedaldi. Invariant information clustering for unsupervised image classification and segmentation. In ICCV, 2019. 17, 18

    2019

  20. [28]

    Billion-scale similarity search with gpus

    Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. IEEE Transactions on Big Data ,

  21. [29]

    Co- tracker3: Simpler and better point tracking by pseudo- labelling real videos

    Nikita Karaev, Iurii Makarov, Jianyuan Wang, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Co- tracker3: Simpler and better point tracking by pseudo- labelling real videos. arXiv preprint arXiv:2410.11831, 2024. 3, 5

    2024 arXiv

  22. [30]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2014. 14

    2014

  23. [31]

    Panoptic segmentation

    Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollar. Panoptic segmentation. In CVPR,

  24. [32]

    Principles of Gestalt psychology

    Kurt Koffka. Principles of Gestalt psychology . routledge,

  25. [33]

    The hungarian method for the assignment problem

    Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly , 1955. 5, 6, 7, 15, 16, 17 10

    1955

  26. [34]

    Tracktention: Leveraging point tracking to attend videos faster and better.arXiv preprint arXiv:2503.19904, 2025

    Zihang Lai and Andrea Vedaldi. Tracktention: Leveraging point tracking to attend videos faster and better.arXiv preprint arXiv:2503.19904, 2025. 2

    2025 arXiv

  27. [35]

    Smooseg: smoothness prior for unsupervised semantic segmentation

    Mengcheng Lan, Xinjiang Wang, Yiping Ke, Jiaxing Xu, Litong Feng, and Wayne Zhang. Smooseg: smoothness prior for unsupervised semantic segmentation. NeurIPS, 2023. 2

    2023

  28. [36]

    Cribo: Self-supervised learning via cross-image object-level bootstrapping

    Tim Lebailly, Thomas Stegmüller, Behzad Bozorgtabar, Jean- Philippe Thiran, and Tinne Tuytelaars. Cribo: Self-supervised learning via cross-image object-level bootstrapping. In ICLR,

  29. [37]

    Joint-task self-supervised learning for temporal correspondence

    Xueting Li, Sifei Liu, Shalini De Mello, Xiaolong Wang, Jan Kautz, and Ming-Hsuan Yang. Joint-task self-supervised learning for temporal correspondence. In NeurIPS, 2019. 2

    2019

  30. [38]

    Exploring plain vision transformer backbones for object de- tection

    Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object de- tection. In ECCV, 2022. 13

    2022

  31. [39]

    Cross pixel optical-flow similarity for self-supervised learn- ing

    Aravindh Mahendran, James Thewlis, and Andrea Vedaldi. Cross pixel optical-flow similarity for self-supervised learn- ing. In ACCV, 2019. 2

    2019

  32. [40]

    Deep spectral methods: A surprisingly strong baseline for unsupervised semantic segmentation and local- ization

    Luke Melas-Kyriazi, Christian Rupprecht, Iro Laina, and An- drea Vedaldi. Deep spectral methods: A surprisingly strong baseline for unsupervised semantic segmentation and local- ization. In CVPR, 2022. 7, 15

    2022

  33. [41]

    You don’t need domain- specific data augmentations when scaling self-supervised learning

    Théo Moutakanni, Maxime Oquab, Marc Szafraniec, Maria Vakalopoulou, and Piotr Bojanowski. You don’t need domain- specific data augmentations when scaling self-supervised learning. NeurIPS, 2025. 3

    2025

  34. [42]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. TMLR,

  35. [43]

    Hummingbird evaluation for vision encoders, 2024

    Valentinos Pariza, Mohammadreza Salehi, and Yuki Asano. Hummingbird evaluation for vision encoders, 2024. 16

    2024

  36. [44]

    Burgh- outs, Francesco Locatello, and Yuki M Asano

    Valentinos Pariza, Mohammadreza Salehi, Gertjan J. Burgh- outs, Francesco Locatello, and Yuki M Asano. Near, far: Patch-ordering enhances vision foundation models’ scene un- derstanding. In The Thirteenth International Conference on Learning Representations, 2025. 2

    2025

  37. [45]

    Self-supervised video pretraining yields human- aligned visual representations

    Nikhil Parthasarathy, SM Eslami, João Carreira, and Olivier J Hénaff. Self-supervised video pretraining yields human- aligned visual representations. In NeurIPS, 2023. 2

    2023

  38. [46]

    Pytorch: An imperative style, high-performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zem- ing Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. NeurIPS, 2019. 13

    2019

  39. [47]

    Learning features by watching objects move

    Deepak Pathak, Ross Girshick, Piotr Dollár, Trevor Darrell, and Bharath Hariharan. Learning features by watching objects move. In CVPR, 2017. 2

    2017

  40. [48]

    The 2017 davis challenge on video object segmentation

    Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Ar- beláez, Alex Sorkine-Hornung, and Luc Van Gool. The 2017 davis challenge on video object segmentation. arXiv preprint arXiv:1704.00675, 2017. 5, 6, 16, 18

    2017 arXiv

  41. [49]

    Imagenet large scale visual recognition challenge

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, San- jeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. IJCV, 2015. 14

    2015

  42. [50]

    Time does tell: Self-supervised time- tuning of dense image representations

    Mohammadreza Salehi, Efstratios Gavves, Cees GM Snoek, and Yuki M Asano. Time does tell: Self-supervised time- tuning of dense image representations. In ICCV, 2023. 1, 2, 3, 5, 6, 7, 15, 16, 21

    2023

  43. [51]

    Sigma: Sinkhorn-guided masked video modeling

    Mohammadreza Salehi, Michael Dorkenwald, Fida Moham- mad Thoker, Efstratios Gavves, Cees GM Snoek, and Yuki M Asano. Sigma: Sinkhorn-guided masked video modeling. In ECCV, 2024. 4

    2024

  44. [52]

    Bridging the gap to real-world object-centric learning

    Maximilian Seitzer, Max Horn, Andrii Zadaianchuk, Dominik Zietlow, Tianjun Xiao, Carl-Johann Simon-Gabriel, Tong He, Zheng Zhang, Bernhard Schölkopf, Thomas Brox, et al. Bridging the gap to real-world object-centric learning. In ICLR, 2023. 2, 7, 15

    2023

  45. [53]

    Unsupervised object local- ization: Observing the background to discover objects

    Oriane Siméoni, Chloé Sekkat, Gilles Puy, Antonín V obeck`y, Éloi Zablocki, and Patrick Pérez. Unsupervised object local- ization: Observing the background to discover objects. In CVPR, 2023. 2

    2023

  46. [54]

    Croc: Cross-view on- line clustering for dense visual representation learning

    Thomas Stegmüller, Tim Lebailly, Behzad Bozorgtabar, Tinne Tuytelaars, and Jean-Philippe Thiran. Croc: Cross-view on- line clustering for dense visual representation learning. In CVPR, 2023. 2, 6, 7, 15, 16

    2023

  47. [55]

    Segmenter: Transformer for semantic segmentation

    Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In ICCV, 2021. 5, 13, 14

    2021

  48. [56]

    Eva-clip: Improved training techniques for clip at scale

    Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389, 2023. 7, 15, 16

    2023 arXiv

  49. [57]

    Video- mae: Masked autoencoders are data-efficient learners for self-supervised video pre-training

    Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Video- mae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. NeurIPS, 2022. 1

    2022

  50. [58]

    Self-supervised learning of video-induced visual invariances

    Michael Tschannen, Josip Djolonga, Marvin Ritter, Aravindh Mahendran, Neil Houlsby, Sylvain Gelly, and Mario Lucic. Self-supervised learning of video-induced visual invariances. In CVPR, 2020. 2

    2020

  51. [59]

    Unsupervised semantic segmenta- tion by contrasting object mask proposals

    Wouter Van Gansbeke, Simon Vandenhende, Stamatios Geor- goulis, and Luc Van Gool. Unsupervised semantic segmenta- tion by contrasting object mask proposals. In ICCV, 2021. 1, 7, 15, 17

    2021

  52. [60]

    Moving off-the- grid: Scene-grounded video representations

    Sjoerd van Steenkiste, Daniel Zoran, Yi Yang, Yulia Rubanova, Rishabh Kabra, Carl Doersch, Dilara Gokay, Eti- enne Pot, Klaus Greff, Drew Hudson, et al. Moving off-the- grid: Scene-grounded video representations. NeurIPS, 2025. 2

    2025

  53. [61]

    Is imagenet worth 1 video? learning strong image encoders from 1 long unlabelled video

    Shashanka Venkataramanan, Mamshad Nayeem Rizve, João Carreira, Yuki M Asano, and Yannis Avrithis. Is imagenet worth 1 video? learning strong image encoders from 1 long unlabelled video. In ICLR, 2024. 1, 2

    2024

  54. [62]

    Unsupervised learning of visual representations using videos

    Xiaolong Wang and Abhinav Gupta. Unsupervised learning of visual representations using videos. In ICCV, 2015. 2

    2015

  55. [63]

    Learning correspondence from the cycle-consistency of time

    Xiaolong Wang, Allan Jabri, and Alexei A Efros. Learning correspondence from the cycle-consistency of time. In CVPR,

  56. [64]

    Self-supervised representation learning from flow equivariance

    Yuwen Xiong, Mengye Ren, Wenyuan Zeng, and Raquel Urtasun. Self-supervised representation learning from flow equivariance. In ICCV, 2021. 2

    2021

  57. [65]

    Youtube-vos: 11 A large-scale video object segmentation benchmark

    Ning Xu, Linjie Yang, Yuchen Fan, Dingcheng Yue, Yuchen Liang, Jianchao Yang, and Thomas Huang. Youtube-vos: 11 A large-scale video object segmentation benchmark. arXiv preprint arXiv:1809.03327, 2018. 5, 6, 16, 18

    2018 arXiv

  58. [66]

    Patch-level representation learning for self-supervised vision transformers

    Sukmin Yun, Hankook Lee, Jaehyung Kim, and Jinwoo Shin. Patch-level representation learning for self-supervised vision transformers. In CVPR, 2022. 6, 15

    2022

  59. [67]

    Unsupervised se- mantic segmentation with self-supervised object-centric rep- resentations

    Andrii Zadaianchuk, Matthaeus Kleindessner, Yi Zhu, Francesco Locatello, and Thomas Brox. Unsupervised se- mantic segmentation with self-supervised object-centric rep- resentations. In ICLR, 2023. 7, 15

    2023

  60. [68]

    Object-centric learning for real-world videos by predicting temporal feature similarities

    Andrii Zadaianchuk, Maximilian Seitzer, and Georg Martius. Object-centric learning for real-world videos by predicting temporal feature similarities. NeurIPS, 2023. 2

    2023

  61. [69]

    Scene parsing through ade20k dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Bar- riuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, 2017. 5, 6, 15, 17, 18

    2017

  62. [70]

    ibot: Image bert pre-training with online tokenizer

    Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. In ICLR, 2022. 7, 15, 16

    2022

  63. [71]

    Self-supervised learning of object parts for semantic segmentation

    Adrian Ziegler and Yuki M Asano. Self-supervised learning of object parts for semantic segmentation. In CVPR, 2022. 1, 2, 6, 7, 8, 15, 16, 17 12

    2022

  64. [74]

    Additional Experiments Comparison to TimeT

    Appendix 7.1. Additional Experiments Comparison to TimeT. Here, we compare MoSiC with TimeT, both initialized from the same DINO backbone. As shown in Figure 5, MoSiC consistently outperforms TimeT across Pascal VOC and ADE20K for both DINO and DI- NOv2 backbones. Notably, whi...

  65. [75]

    Memory Bank Construction: Given a dataset of images with dense annotations, two memory banks are created. One stores image patch features extracted from the spatial output of a dense encoder applied to the training images, while the other stores the corresponding patch labels ...

  66. [76]

    Unsupervised video semantic segmentation results for clustering and over-clustering on DA VIS [48] and Youtube-VOS (YTVOS) [65]

    Query Processing: For each image in the validation set, 15 Table 12. Unsupervised video semantic segmentation results for clustering and over-clustering on DA VIS [48] and Youtube-VOS (YTVOS) [65]. For clustering, the Hungarian algorithm [33] matches clusters (K) to ground tru...

  67. [77]

    Since the original implementation by [5] is unavailable, we use the open-source implementation from [ 43]

    Comparison: The generated annotation for the image is compared against the ground truth annotation to evaluate performance. Since the original implementation by [5] is unavailable, we use the open-source implementation from [ 43]. This im- plementation adheres to the original ...

    2012

  68. [78]

    stuff" categories and 80

    Dataset Details 8.1. Image Datasets Pascal VOC 2012[18] This dataset, using the latest trainaug split, consists of 10,582 images with annotations spanning 21 classes, including one background class. The validation set contains 1,449 images. Following [59], we ignore unlabeled ...

    2012

  69. [2021]

    2, 5, 6, 7, 8, 14, 15, 16

  70. [2024]

    2, 4, 5, 6, 7, 15, 16

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.