Pith. sign in

REVIEW 3 major objections 5 minor 70 references

Reviewed by Pith at T0; open to challenge.

T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

Parts, not pixels: tracking objects in video without labels

2026-07-09 16:35 UTC pith:Z4PMQVT3

load-bearing objection Solid engineering, but the backbone confound is the real issue the 3 major comments →

arxiv 2607.07230 v1 pith:Z4PMQVT3 submitted 2026-07-08 cs.CV

`Attention-Guided Cross-Temporal Clustering for Self-Supervised Video Object Segmentation

Waqas Arshid , Mohammad Awrangjeb , Alan Wee-Chung Liew , Yongsheng Gao This is my paper
classification cs.CV
keywords acrossclusteringself-supervisedtemporalvideoattention-guidedconsistencycross-temporal
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper proposes that video object segmentation can be done without any labels by tracking object *parts*—not whole objects and not individual pixels—across time. The method takes a frozen vision transformer (SAM2) and uses its built-in attention map to pick out the most semantically meaningful patches in each frame. A tiny clustering head then assigns each patch a soft distribution over a small number of latent parts (like wheels, ears, or wings). The core mechanism for learning is a symmetric consistency loss: if a patch in frame t matches a patch in frame t+3, their part distributions should agree, and this agreement is weighted by how salient each patch is. By enforcing this at multiple temporal offsets simultaneously, the system learns temporally stable part groupings without optical flow, synthetic motion, or annotated masks. The paper claims this mid-level abstraction—parts that persist through occlusion and deformation—gives better temporal coherence than pixel-level correspondence while running at real-time speed on a single GPU.

Core claim

The central claim is that part-level temporal alignment, operationalized as symmetric KL divergence between soft cluster assignments of attention-selected tokens across frames, is a sufficient learning signal for self-supervised video object segmentation. The paper shows that this formulation avoids the fragility of optical flow under fast motion and the computational cost of dense pixel-level correspondence, while producing clusters that are temporally stable and transferable across datasets. The method achieves competitive scores on DAVIS-2017, DAVIS-2016, and YouTube-VOS benchmarks under a zero-shot protocol, with the clustering head contributing only ~0.04 GFLOPs atop a frozen backbone.

What carries the argument

The method has four components: (1) a frozen SAM2 ViT encoder that produces dense token embeddings and a [CLS]-attention saliency prior; (2) an adaptive token selector that retains the top-p fraction of attention mass while enforcing B×B grid diversity to prevent spatial collapse; (3) a two-layer MLP that maps each selected token to a K-dimensional soft part distribution; (4) multi-offset temporal matching via mutual nearest-neighbor cosine similarity at strides {1,2,4,8}, filtered by a match-rate controller, with a saliency-weighted symmetric KL loss aligning part distributions across matched pairs. Two regularizers—an entropy penalty to sharpen assignments and a KL-to-uniform penalty to平衡化

Load-bearing premise

The [CLS] attention map from the frozen SAM2 encoder is assumed to be a reliable indicator of semantically meaningful regions across diverse video domains. The paper itself acknowledges this signal can be unreliable for small objects, in cluttered scenes, or under domain shift. If this attention prior is systematically biased toward large or high-contrast regions, the entire pipeline—token selection, saliency-weighted loss, and part discovery—would inherit that bias.

What would settle it

If one were to replace the [CLS]-attention saliency prior with a random or uniform weighting and the method still achieved comparable performance, the attention-guided token selection would not be load-bearing. Conversely, if performance collapsed disproportionately for small objects or cluttered scenes compared to prior flow-based methods, the part-level approach would be shown to inherit the saliency prior's weaknesses rather than mitigating them.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If part-level alignment is sufficient for VOS without labels, the same principle could transfer to other temporal understanding tasks like action segmentation or scene graph prediction in video, where persistent mid-level structure matters more than pixel-precise correspondence.
  • The finding that a frozen backbone's attention map provides a usable saliency prior suggests that other large pretrained models with different attention structures could serve as drop-in replacements, potentially yielding different part discovery patterns without retraining the encoder.
  • The match-rate controller that filters temporal offsets by correspondence reliability could be adopted as a general mechanism for any self-supervised temporal learning system to avoid training on noisy matches.
  • The demonstration that only ~0.04 GFLOPs of trainable parameters atop a frozen backbone yields competitive results may shift the field toward lightweight adapter-style architectures for video tasks rather than end-to-end fine-tuning.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The manuscript proposes CTC2, a self-supervised framework for video object segmentation (VOS) that discovers temporally consistent, part-level representations. The method uses a frozen SAM2 ViT encoder to extract spatial tokens and a [CLS] attention-derived saliency prior. An adaptive top-p token selection strategy with grid diversity retains a compact, informative subset of tokens. A lightweight MLP clustering head maps these tokens to soft part distributions, which are aligned across frames using a saliency-weighted symmetric KL divergence loss over mutual nearest-neighbor correspondences. The framework includes multi-offset temporal supervision with match-rate control and regularization to prevent cluster collapse. Evaluations on DAVIS-2016, DAVIS-2017, and YouTube-VOS show competitive performance under a zero-shot protocol, with additional ablation studies, cross-dataset generalization tests, and semi-supervised results provided.

Significance. The paper addresses a relevant problem in self-supervised VOS by proposing a mid-level, part-centric formulation that avoids optical flow, dense warping, or external memory modules. The use of attention-guided token selection to enforce temporal consistency on a compact subset of tokens is a practical and efficient design choice. The authors provide custom metrics (Temporal Part Stability and Cluster Identity Retention) to quantitatively evaluate the temporal coherence of discovered parts, which is a welcome addition beyond standard VOS benchmarks. The efficiency analysis (Table 13) transparently isolating the compute cost of the frozen backbone versus the proposed clustering head is also a strength.

major comments (3)
  1. The central claim—that part-level temporal consistency via the proposed clustering and symmetric KL loss drives the competitive VOS performance—is not sufficiently isolated from the contribution of the frozen SAM2 backbone. The baselines compared in Table 2 (e.g., TripleNet, CorrFlow, TimeCycle) use older and weaker backbones. The performance gap may be largely attributable to the representation power of SAM2 rather than the proposed temporal consistency mechanism. The ablations in Tables 5-7 only show marginal gains (ΔG ≈ 0.003 to 0.008) for individual components. A critical missing ablation is training the clustering head with only the regularization losses (L_conf + L_bal) and setting the temporal consistency loss (L_cons) weight to 0 in Eq. (15). If performance remains similar without L_cons, the central claim is undermined. This control experiment is load-bearing for the paper's核心贡献
  2. The Temporal Part Stability (TPS) metric defined in Eq. (16) is structurally 1 minus the symmetric KL divergence that the method optimizes in Eq. (9). Consequently, Table 8, which shows that the symmetric KL loss yields higher TPS than one-sided KL or cross-entropy, is partially tautological. While the Cluster Identity Retention (CIR) metric in Table 9 is non-circular and provides some evidence for the stability of hard part labels, the reliance on TPS as a primary validation metric for the loss function is problematic. The authors should either remove the TPS analysis or clearly acknowledge its structural dependency on the optimized objective, focusing the evaluation on non-circular metrics like CIR.
  3. The evaluation protocol fixes the cluster-to-instance permutation using the first annotated frame via Hungarian matching (Sec. 4). The authors acknowledge this 'may partially obscure temporal identity drift in long sequences' (Sec. 4, Sec. 5.3). This is a significant limitation for a method whose central goal is temporal consistency. It is unclear how much identity drift occurs after the first frame. The paper would be substantially strengthened by reporting a metric that does not rely on a first-frame anchor, or by providing quantitative evidence of drift over time (e.g., CIR evaluated at longer intervals without re-anchoring) to demonstrate that the temporal consistency mechanism genuinely prevents drift rather than masking it.
minor comments (5)
  1. Sec. 3.4, Eq. (2): The parameter p is stated to be in the range [0.80, 0.90], but the experimental setup (Sec. 4) fixes p=0.85. The rationale for this specific value and the sensitivity of the method to this choice are not discussed.
  2. Table 2: The method 'BA [21]' is listed with F=0.486, which is higher than its J=0.392, but the overall G=0.439. This is mathematically correct, but the text in Sec. 5.1 states that CTC2 achieves gains 'consistent across both region similarity and boundary accuracy.' It would be helpful to explicitly note that CTC2 outperforms BA on both J and F, as BA's high F-score is a notable baseline.
  3. Sec. 5.5, Table 10: The cross-dataset result for CTC2 trained on DAVIS and evaluated on YouTube-VOS shows G=0.547, which is reported as ΔG = -0.023. However, the in-domain YouTube-VOS score is G=0.570. The calculation 0.547 - 0.570 = -0.023 is correct, but the text states this yields 'no degradation,' which is inaccurate. It is a small degradation, though less severe than baselines. This should be rephrased for precision.
  4. The manuscript uses 'CTC2' and 'CTC²' interchangeably (e.g., Fig. 1 caption vs. main text). Consistency in naming would improve readability.
  5. Sec. 3.6, Eq. (8): The weight w_∆t is defined as proportional to γ^∆t, with γ ∈ [0.6, 0.8]. Sec. 4 states γ is annealed linearly from 0.8 to 0.6. It would be helpful to clarify whether the weights w_∆t are renormalized after annealing γ, as the sum constraint in Eq. (8) must be maintained.

Circularity Check

2 steps flagged

The TPS metric (Eq. 16) is defined as 1 minus the symmetric KL divergence that the method optimizes (Eq. 9), making Table 8 partially tautological; the non-circular CIR metric (Table 9) provides independent support.

specific steps
  1. self definitional [Eq. 16 (Sec. 5.4) and Table 8]
    "TPS(Δt) = 1 − (1/|M_Δt|) Σ_{(i,j)∈M_Δt} (1/2)[KL(C_t^(i) ∥ C_{t+Δt}^(j)) + KL(C_{t+Δt}^(j) ∥ C_t^(i))]."

    The temporal consistency loss L_CTC2 (Eq. 9/11) is defined as the saliency-weighted symmetric KL divergence between matched token pairs: SKL(P_t^(i), P_{t+Δt}^(j)) = (1/2)[KL(P_t^(i)∥P_{t+Δt}^(j)) + KL(P_{t+Δt}^(j)∥P_t^(i))]. The TPS metric (Eq. 16) is defined as 1 minus this exact same symmetric KL divergence averaged over matched pairs. Table 8 then reports that the symmetric KL loss yields higher TPS than one-sided KL or cross-entropy. This is partially circular: the method optimizes symmetric KL, and TPS measures 1 minus symmetric KL, so the loss that directly minimizes symmetric KL will mechanically produce higher TPS by construction. The comparison in Table 8 is not fully tautological (the loss is saliency-weighted while TPS is uniform, and they are computed on different data subsets

  2. self definitional [Eq. 9 (Sec. 3.7) vs. Eq. 16 (Sec. 5.4)]
    "SKL(P_t^(i), P_{t+Δt}^(j)) = (1/2)[KL(P_t^(i)∥P_{t+Δt}^(j)) + KL(P_{t+Δt}^(j)∥P_t^(i))] ... TPS(Δt) = 1 − (1/|M_Δt|) Σ_{(i,j)∈M_Δt} (1/2)[KL(C_t^(i)∥C_{t+Δt}^(j)) + KL(C_{t+Δt}^(j)∥C_t^(i))]"

    The structural form of the evaluation metric TPS is identical to the training objective L_CTC2 (minus the saliency weighting ω_ij). When the paper states in Table 8 that 'The symmetric KL objective consistently yields higher TPS than one-sided KL or cross-entropy losses,' this result is expected by construction for the symmetric KL component of TPS: a model trained to minimize symmetric KL will naturally have lower symmetric KL on held-out data than a model trained with one-sided KL or cross-entropy. The paper does not acknowledge this structural identity. However, the CIR metric (Table 9, Eq. 17) uses hard argmax labels and is not structurally identical to the loss, providing non-circular corroborating evidence. The main VOS results (Tables 2-4) use external benchmarks and are not self-re

full rationale

The paper's central VOS performance claims (Tables 2-4) are evaluated on external benchmarks (DAVIS, YouTube-VOS) using standard metrics (J, F, G) and are not circular. The frozen SAM2 backbone provides external representation capacity independent of the paper's contributions. The component ablations (Tables 5-7) compare design variants on the same backbone, providing independent evidence. The one circularity found is in the TPS metric (Eq. 16, Table 8): TPS is defined as 1 minus the symmetric KL divergence, which is the same symmetric KL divergence that the method's loss function (Eq. 9) directly optimizes. Showing that the symmetric KL loss yields higher TPS than alternative losses is therefore partially tautological—a model trained to minimize symmetric KL will naturally produce lower symmetric KL (higher TPS) than models trained with different objectives. This is a minor circularity because (1) the loss includes saliency weighting while TPS does not, creating a slight structural difference, (2) the non-circular CIR metric (Table 9) provides corroborating evidence using hard labels rather than the loss function's form, and (3) the main performance claims do not depend on TPS. The paper would have been stronger had it acknowledged this structural identity and relied more heavily on CIR, but the circularity is localized to one validation table rather than undermining the central claims.

Axiom & Free-Parameter Ledger

9 free parameters · 4 axioms · 2 invented entities

The framework introduces 9 free parameters tuned by validation, relies on 4 domain/ad-hoc axioms (most notably the [CLS] attention saliency prior and the first-frame permutation evaluation protocol), and defines 2 self-referential metrics. The parameter count is moderate for a method paper, but the evaluation protocol axiom is a notable source of potential bias.

free parameters (9)
  • p (saliency mass threshold) = 0.85
    Controls retained saliency mass in top-p selection; chosen by validation.
  • k_min, k_max (token budget bounds) = 24, 128
    Hard bounds on adaptive token budget; chosen empirically.
  • B (grid diversity cells) = 4
    Controls spatial diversity grid partitioning.
  • K (number of part prototypes) = 16
    Dimension of soft part distribution; selected via validation.
  • delta (similarity threshold) = 0.4
    Threshold for mutual nearest-neighbor filtering.
  • r_min (match-rate threshold) = 0.6
    Controls which temporal offsets are retained.
  • gamma (offset decay) = 0.8 to 0.6
    Annealed linearly during training; weights temporal offsets.
  • lambda_conf, lambda_bal (regularization weights) = 0.1, 1.0
    Balance entropy sharpening and cluster usage regularization.
  • d_h (MLP hidden dimension) = 512
    Hidden layer width of clustering head.
axioms (4)
  • domain assumption [CLS] attention from a frozen ViT correlates with semantically meaningful object and part regions.
    Invoked in Sec. 3.2 and Sec. 3.3 to justify using attention as a saliency prior. The paper acknowledges this is an 'imperfect heuristic' (Sec. 3.2).
  • domain assumption Soft part assignments should remain temporally consistent across frames under motion and occlusion.
    Core inductive bias stated in Sec. 1 and Sec. 3.1; drives the symmetric KL consistency objective.
  • domain assumption Mutual nearest-neighbor matching in embedding space yields reliable temporal correspondences.
    Used in Sec. 3.6 (Eq. 5) to establish temporal alignment; standard in correspondence-based SSL but assumes embedding space is metric.
  • ad hoc to paper Fixing cluster-to-instance permutation from the first frame is a valid evaluation protocol for unsupervised VOS.
    Stated in Sec. 4; the paper admits this 'may partially obscure temporal identity drift' (Sec. 4), introducing potential evaluation bias.
invented entities (2)
  • Temporal Part Stability (TPS) metric no independent evidence
    purpose: Measures consistency of soft part assignments across time for matched token pairs.
    Self-defined metric (Eq. 16) that directly measures the quantity optimized by the symmetric KL loss; used in Sec. 5.4 to validate the loss design.
  • Cluster Identity Retention metric no independent evidence
    purpose: Evaluates hard label consistency for matched tokens across time.
    Self-defined metric (Eq. 17); supplementary to TPS and also self-referential to the clustering objective.

pith-pipeline@v1.1.0-glm · 26235 in / 2396 out tokens · 437461 ms · 2026-07-09T16:35:01.337911+00:00 · methodology

0 comments
read the original abstract

Video object segmentation (VOS) is a fundamental task in video understanding, requiring accurate delineation and consistent tracking of objects across frames. While supervised methods achieve strong performance, they rely on densely annotated datasets that are costly to obtain and have limited domain coverage. Self-supervised learning offers a promising alternative by removing the need for manual labels; however, existing approaches often struggle to jointly maintain spatial accuracy and temporal coherence, particularly in unconstrained multi-object scenarios. Many rely on optical flow, synthetic motion cues, or task-specific pretraining, limiting scalability and generalisation. We propose a self-supervised framework, Cross-Temporal Consistency and Clustering, that learns mid-level, part-aware representations by combining attention-guided token selection with lightweight temporal clustering. Instead of operating at the pixel or whole-object level, the method aligns soft part assignments across time using a saliency-weighted symmetric consistency objective. The framework leverages a frozen transformer backbone with lightweight modules for adaptive token selection and multi-offset temporal alignment, enabling efficient scaling across resolutions and motion patterns.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

70 extracted references · 70 canonical work pages · 12 internal anchors

  1. [1]

    Discriminative Spatial-Semantic VOS Solution: 1st Place Solution for 6th LSVOS

    Miao, D., Gu, Y., Li, X., He, Z., Wang, Y., Yang, M.-H.: Discriminative spatial-semantic vos solution: 1st place solution for 6th lsvos. arXiv preprint arXiv:2408.16431 (2024) https://doi.org/10.48550/arXiv.2408.16431

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2408.16431 2024

  2. [2]

    In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016)

    Perazzi, F., Pont-Tuset, J., McWilliams, B., Van Gool, L., Gross, M., Sorkine- Hornung, A.: A benchmark dataset and evaluation methodology for video object segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016). https://doi.org/10.1109/cvpr.2016.85

    work page doi:10.1109/cvpr.2016.85 2016

  3. [3]

    In: Proceedings of the European Conference on Computer Vision (2018)

    Xu, N., Yang, L., Fan, Y., Yang, J., Yue, D., Liang, Y., Price, B., Cohen, S., Huang, T.: Youtube-vos: Sequence-to-sequence video object segmentation. In: Proceedings of the European Conference on Computer Vision (2018). https:// doi.org/10.1007/978-3-030-01228-1_36

    work page doi:10.1007/978-3-030-01228-1_36 2018

  4. [4]

    IEEE Transactions on Pattern Analysis and Machine Intelligence (2018) https://doi.org/10.1109/TPAMI.2018.2838670

    Maninis, K.-K., Caelles, S., Chen, Y., Van Gool, L.: Video object segmenta- tion without temporal information. IEEE Transactions on Pattern Analysis and Machine Intelligence (2018) https://doi.org/10.1109/TPAMI.2018.2838670

    work page doi:10.1109/tpami.2018.2838670 2018

  5. [5]

    Self-supervised Learning for Video Correspondence Flow

    Lai, Z., Xie, W.: Self-supervised learning for video correspondence flow. arXiv preprint arXiv:1905.00875 (2019) https://doi.org/10.48550/arXiv.1905.00875

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.1905.00875 1905

  6. [6]

    In: Proceedings of the European Conference on Computer Vision (2018)

    Vondrick, C., Shrivastava, A., Fathi, A., Guadarrama, S., Murphy, K.: Tracking emerges by colorizing videos. In: Proceedings of the European Conference on Computer Vision (2018). https://doi.org/10.1007/978-3-030-01261-8_24

    work page doi:10.1007/978-3-030-01261-8_24 2018

  7. [7]

    Berg, Wan-Yen Lo, Piotr Doll ´ar, and Ross B

    Salehi, M., Gavves, E., Snoek, C.G., Asano, Y.M.: Time does tell: Self-supervised time-tuning of dense image representations. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (2023). https://doi.org/10.1109/ iccv51070.2023.01516

    work page arXiv 2023

  8. [8]

    In: Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (2019)

    Wang, X., Jabri, A., Efros, A.A.: Learning correspondence from the cycle- consistency of time. In: Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (2019). https://doi.org/10.1109/cvpr.2019. 00267

    work page doi:10.1109/cvpr.2019 2019

  9. [9]

    IEEE Transactions on Pattern Analysis and Machine Intelligence (2021) https://doi.org/10.1007/978-3-030-58558-7_20

    Yang, Z., Wei, Y., Yang, Y.: Collaborative video object segmentation by multi- scale foreground-background integration. IEEE Transactions on Pattern Analysis and Machine Intelligence (2021) https://doi.org/10.1007/978-3-030-58558-7_20

    work page doi:10.1007/978-3-030-58558-7_20 2021

  10. [10]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference (2025)

    Huang, N., Zheng, W., Xu, C., Keutzer, K., Zhang, S., Kanazawa, A., Wang, Q.: Segment any motion in videos. In: Proceedings of the Computer Vision and Pattern Recognition Conference (2025). https://doi.org/10.1109/cvpr52734. 2025.00323

    work page doi:10.1109/cvpr52734 2025

  11. [11]

    Tracking Anything in High Quality

    Zhu, J., Chen, Z., Hao, Z., Chang, S., Zhang, L., Wang, D., Lu, H., Luo, B., 28 He, J.-Y., Lan, J.-P.,et al.: Tracking anything in high quality. arXiv preprint arXiv:2307.13974 (2023) https://doi.org/10.48550/arXiv.2307.13974

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2307.13974 2023

  12. [12]

    In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Ziegler, A., Asano, Y.M.: Self-supervised learning of object parts for seman- tic segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14502–14511 (2022). https://doi.org/10. 1109/cvpr52688.2022.01410

    work page arXiv 2022

  13. [13]

    In: European Conference on Computer Vision (2024)

    Wei, Y., Gupta, A., Morgado, P.: Towards latent masked image modelling for self- supervised visual representation learning. In: European Conference on Computer Vision (2024). https://doi.org/10.1007/978-3-031-72933-1_1

    work page doi:10.1007/978-3-031-72933-1_1 2024

  14. [14]

    IEEE Transactions on Image Processing (2024) https://doi.org/10.1109/tip.2024.3374199

    Zheng, X., Liao, L., Jiao, J., Gao, F., Wang, R.: Surface-sos: Self-supervised object segmentation via neural surface representation. IEEE Transactions on Image Processing (2024) https://doi.org/10.1109/tip.2024.3374199

    work page doi:10.1109/tip.2024.3374199 2024

  15. [15]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Hung, W.-C., Jampani, V., Liu, S., Molchanov, P., Yang, M.-H., Kautz, J.: Scops: Self-supervised co-part segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 869–878 (2019). https://doi.org/10.1109/cvpr.2019.00096

    work page doi:10.1109/cvpr.2019.00096 2019

  16. [16]

    In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Xu, M., Zhang, Z., Zhang, H., Wei, Y., Xie, E., Li, Z., Loy, C.C.: Groupvit: Semantic segmentation emerges from text supervision. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2022). https://doi.org/10.1109/cvpr52688.2022.01760

    work page doi:10.1109/cvpr52688.2022.01760 2022

  17. [17]

    Pattern Recognition (2010) https://doi.org/10.1016/j.patcog.2009.08.005

    Kim, H.Y.: Rotation-discriminating template matching based on fourier coef- ficients of radial projections with robustness to scaling and partial occlusion. Pattern Recognition (2010) https://doi.org/10.1016/j.patcog.2009.08.005

    work page doi:10.1016/j.patcog.2009.08.005 2010

  18. [19]

    10 João Carreira, Dilara Gokay, Michael King, Chuhan Zhang, Ignacio Rocco, Aravindh Mahendran, Thomas Albert Keck, et al

    Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (2021). https://doi.org/10.1109/ICCV48922.2021.00951

    work page doi:10.1109/iccv48922.2021.00951 2021

  19. [20]

    Segment Anything

    Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, L., Gustafson, J., Xiao, T., Whitehead, S., Cottrell, R., Dolson, J.,et al.: Segment anything. arXiv preprint arXiv:2304.02643 (2023) https://doi.org/10.1109/ICCV51070.2023.00371

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.1109/iccv51070.2023.00371 2023

  20. [21]

    In: European Conference on Computer Vision (2024)

    Ding, S., Qian, R., Xu, H., Lin, D., Xiong, H.: Betrayed by attention: A simple yet effective approach for self-supervised video object segmentation. In: European Conference on Computer Vision (2024). https://doi.org/10.1007/ 29 978-3-031-72995-9_13

    work page 2024

  21. [22]

    In: Proceedings of the European Conference on Computer Vision (2022)

    Cheng, S., Wang, J., Shen, Y., Joo, J., Zhang, X.: Xmem: Long-term video object segmentation with an atkinson-shiffrin memory model. In: Proceedings of the European Conference on Computer Vision (2022). https://doi.org/10.1007/ 978-3-031-19815-1_37

    work page 2022

  22. [23]

    In: Advances in Neural Information Processing Systems, vol

    Yang, Z., Wei, Y., Yang, Y.: Associating objects with transformers for video object segmentation. In: Advances in Neural Information Processing Systems, vol. 34, pp. 2491–2502 (2021)

    work page 2021

  23. [24]

    The 2017 DAVIS Challenge on Video Object Segmentation

    Pont-Tuset, J., Perazzi, F., Caelles, S., Arbelaez, P., Sorkine-Hornung, A., Van Gool, L.: The 2017 davis challenge on video object segmentation. arXiv preprint arXiv:1704.00675 (2017) https://doi.org/10.48550/arXiv.1704.00675

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.1704.00675 2017

  24. [25]

    IEEE transactions on pattern analysis and machine intelligence (2020) https://doi.org/10.1109/tpami.2020.2992393

    Jing, L., Tian, Y.: Self-supervised visual feature learning with deep neural net- works: A survey. IEEE transactions on pattern analysis and machine intelligence (2020) https://doi.org/10.1109/tpami.2020.2992393

    work page doi:10.1109/tpami.2020.2992393 2020

  25. [26]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2020)

    He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum contrast for unsu- pervised visual representation learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2020). https://doi. org/10.20944/preprints202501.0668.v1

    work page doi:10.20944/preprints202501.0668.v1 2020

  26. [27]

    In: Advances in Neural Information Processing Systems, vol

    Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P.H., Buchatskaya, E., Doersch, C., Avila Pires, B., Guo, Z.D., Azar, M.,et al.: Bootstrap your own latent: A new approach to self-supervised learning. In: Advances in Neural Information Processing Systems, vol. 33, pp. 21271–21284 (2020)

    work page 2020

  27. [28]

    Multi-Modal Fusion Transformer for End-to-End Autonomous Driving,

    Chen, X., He, K.: Exploring simple siamese representation learning. In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2021). https://doi.org/10.1109/cvpr46437.2021.01549

    work page doi:10.1109/cvpr46437.2021.01549 2021

  28. [29]

    Multi-Modal Fusion Transformer for End-to-End Autonomous Driving,

    Wang, X., Xie, S., Gupta, A.: Dense contrastive learning for self-supervised visual pre-training. In: CVPR (2021). https://doi.org/10.1109/cvpr46437.2021.00304

    work page doi:10.1109/cvpr46437.2021.00304 2021

  29. [30]

    Multi-Modal Fusion Transformer for End-to-End Autonomous Driving,

    Xie, Z., Lin, Y., Zhang, Z., Cao, Y., Lin, S., Hu, H.: Propagate yourself: Explor- ing pixel-level consistency for unsupervised visual representation learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2021). https://doi.org/10.1109/cvpr46437.2021.01641

    work page doi:10.1109/cvpr46437.2021.01641 2021

  30. [31]

    IEEE transactions on pattern analysis and machine intelligence (2023) https://doi.org/10.1109/tpami.2023.3305122

    Wang,Y.,Shen,X.,Yuan,Y.,Du,Y.,Li,M.,Hu,S.X.,Crowley,J.L.,Vaufreydaz, D.:Tokencut:Segmentingobjectsinimagesandvideoswithself-supervisedtrans- former and normalized cut. IEEE transactions on pattern analysis and machine intelligence (2023) https://doi.org/10.1109/tpami.2023.3305122

    work page doi:10.1109/tpami.2023.3305122 2023

  31. [32]

    Localizing Objects with Self-Supervised Transformers and no Labels

    Siméoni, O., Puy, G., Vo, H.V., Roburin, S., Gidaris, S., Bursuc, A., Pérez, P., 30 Marlet, R., Ponce, J.: Localizing objects with self-supervised transformers and no labels. arXiv preprint arXiv:2109.14279 (2021) https://doi.org/10.5244/c.35.365

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.5244/c.35.365 2021

  32. [33]

    In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Wang, X., Yu, Z., De Mello, S., Kautz, J., Anandkumar, A., Shen, C., Alvarez, J.M.: Freesolo: Learning to segment objects without annotations. In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2022). https://doi.org/10.1109/cvpr52688.2022.01378

    work page doi:10.1109/cvpr52688.2022.01378 2022

  33. [34]

    Cross-dimension affinity distillation for 3d em neuron segmentation,

    Niu, D., Wang, X., Han, X., Lian, L., Herzig, R., Darrell, T.: Unsupervised universal image segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2024). https://doi.org/10.1109/ cvpr52733.2024.02146

    work page arXiv 2024

  34. [35]

    In: Advances in Neural Information Processing Systems, vol

    Araslanov, N., Schaub-Meyer, S., Roth, S.: Dense unsupervised learning for video segmentation. In: Advances in Neural Information Processing Systems, vol. 34, pp. 25308–25319 (2021)

    work page 2021

  35. [36]

    In: Proceedings of the 25th International Conference on Multimodal Interaction (2023)

    Das, K., Abouelenien, M., Burzo, M.G., Elson, J., Prakah-Asante, K., Maranville, C.: Towards autonomous physiological signal extraction from thermal videos using deep learning. In: Proceedings of the 25th International Conference on Multimodal Interaction (2023). https://doi.org/10.1145/3577190.3614123

    work page doi:10.1145/3577190.3614123 2023

  36. [37]

    In: European Conference on Computer Vision (2016)

    Misra, I., Zitnick, C.L., Hebert, M.: Shuffle and learn: unsupervised learning using temporal order verification. In: European Conference on Computer Vision (2016). https://doi.org/10.1007/978-3-319-46448-0_32

    work page doi:10.1007/978-3-319-46448-0_32 2016

  37. [38]

    In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018)

    Wei, D., Lim, J.J., Zisserman, A., Freeman, W.T.: Learning and using the arrow of time. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018). https://doi.org/10.1109/cvpr.2018.00840

    work page doi:10.1109/cvpr.2018.00840 2018

  38. [39]

    In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016)

    Vondrick, C., Pirsiavash, H., Torralba, A.: Anticipating visual representations from unlabeled video. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016). https://doi.org/10.1109/cvpr.2016.18

    work page doi:10.1109/cvpr.2016.18 2016

  39. [41]

    In: Advances in Neural Information Processing Systems, vol

    Kong, Q., Wei, W., Deng, Z., Yoshinaga, T., Murakami, T.: Cyclecontrast for self-supervised video representation learning. In: Advances in Neural Information Processing Systems, vol. 33, pp. 8089–8100 (2020)

    work page 2020

  40. [42]

    In: Advances in Neural Information Processing Systems, vol

    Jabri, A., Owens, A., Efros, A.A.: Space-time correspondence as a contrastive random walk. In: Advances in Neural Information Processing Systems, vol. 33, pp. 19545–19560 (2020) 31

    work page 2020

  41. [43]

    In: Conference on Computer Vision and Pattern Recognition

    Lai, Z., Xie, W., Zisserman, A.: Mast: A memory-augmented self-supervised tracker. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2021). https://doi.org/10.1109/cvpr42600.2020.00651

    work page doi:10.1109/cvpr42600.2020.00651 2021

  42. [44]

    Berg, Wan-Yen Lo, Piotr Doll ´ar, and Ross B

    Doersch, C., Yang, Y., Vecerik, M., Gokay, D., Gupta, A., Aytar, Y., Carreira, J., Zisserman, A.: Tapir: Tracking any point with per-frame initialization and temporal refinement. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (2023). https://doi.org/10.1109/iccv51070.2023.00923

    work page doi:10.1109/iccv51070.2023.00923 2023

  43. [45]

    Deep ViT Features as Dense Visual Descriptors

    Amir, S., Gandelsman, Y., Bagon, S., Dekel, T.: Deep vit features as dense visual descriptors. arXiv preprint arXiv:2112.05814 (2021) https://doi.org/10.48550/ arXiv.2112.05814

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  44. [46]

    In: Advances in Neural Information Processing Systems, vol

    Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Tandon, S., Mensch, A., Milani, S., Huang, P.-S., Han, W.,et al.: Flamingo: a visual language model for few-shot learning. In: Advances in Neural Information Processing Systems, vol. 35, pp. 23716–23736 (2022)

    work page 2022

  45. [47]

    An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

    Dosovitskiy,A.,Beyer,L.,Kolesnikov,A.,Weissenborn,D.,Zhai,X.,Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S.,et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020) https://doi.org/10.48550/arXiv.2010.11929

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2010.11929 2010

  46. [48]

    Representation Learning with Contrastive Predictive Coding

    Oord, A.v.d., Li, Y., Vinyals, O.: Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1807.03748 (2018) https://doi.org/10.48550/ arXiv.1807.03748

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  47. [49]

    In: European Conference on Computer Vision (2024)

    Aubret, A., Teulière, C., Triesch, J.: Self-supervised visual learning from inter- actions with objects. In: European Conference on Computer Vision (2024). https://doi.org/10.1007/978-3-031-73226-3_4

    work page doi:10.1007/978-3-031-73226-3_4 2024

  48. [50]

    In: IEEE Conference on Computer Vision and Pattern Recognition (2024)

    Arshid, W., Awrangjeb, M., Liew, A.W.C., Gao, Y.: Camvos: Leveraging con- text and memory for advanced video object segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition (2024). https://doi.org/10.1109/ dicta63115.2024.00098

    work page arXiv 2024

  49. [51]

    SAM2 for Image and Video Segmentation: A Comprehensive Survey

    Jiaxing, Z., Hao, T.: Sam2 for image and video segmentation: A comprehensive survey. arXiv preprint arXiv:2503.12781 (2025) https://doi.org/10.48550/arXiv. 2503.12781

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv 2025

  50. [52]

    Raghu, M., Unterthiner, T., Kornblith, S., Zhang, C., Dosovitskiy, A.: Do vision transformers see like convolutional neural networks? In: Advances in Neural Information Processing Systems, vol. 34, pp. 12116–12128 (2021)

    work page 2021

  51. [53]

    Token Merging: Your ViT But Faster

    Bolya, D., Fu, C.-Y., Dai, X., Zhang, P., Feichtenhofer, C., Hoffman, J.: Token merging: Your vit but faster. arXiv preprint arXiv:2210.09461 (2022) https:// doi.org/10.48550/arXiv.2210.09461 32

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2210.09461 2022

  52. [54]

    In: Advances in Neural Information Processing Systems, vol

    Rao, Y., Zhao, W., Liu, B., Lu, J., Zhou, J., Hsieh, C.-J.: Dynamicvit: Efficient vision transformers with dynamic token sparsification. In: Advances in Neural Information Processing Systems, vol. 34, pp. 13937–13949 (2021)

    work page 2021

  53. [55]

    Berg, Wan-Yen Lo, Piotr Doll ´ar, and Ross B

    Vasu, P.K.A., Gabriel, J., Zhu, J., Tuzel, O., Ranjan, A.: Fastvit: A fast hybrid vision transformer using structural reparameterization. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (2023). https://doi. org/10.1109/iccv51070.2023.00532

    work page doi:10.1109/iccv51070.2023.00532 2023

  54. [56]

    A critical analysis of self-supervision, or what we can learn from a single image

    Asano, Y.M., Rupprecht, C., Vedaldi, A.: A critical analysis of self-supervision, or what we can learn from a single image. In: International Conference on Learning Representations (ICLR) (2020). https://doi.org/10.48550/arXiv.1904.13132

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.1904.13132 2020

  55. [57]

    In: European Conference on Computer Vision (2020)

    Seong, H., Hyun, J., Kim, E.: Kernelized memory network for video object seg- mentation. In: European Conference on Computer Vision (2020). https://doi. org/10.1007/978-3-030-58542-6_38

    work page doi:10.1007/978-3-030-58542-6_38 2020

  56. [58]

    In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018)

    Sun, D., Yang, X., Liu, M.-Y., Kautz, J.: Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018). https://doi.org/10.1109/cvpr. 2018.00931

    work page doi:10.1109/cvpr 2018

  57. [59]

    In: Proceedings of the European Conference on Computer Vision (2018)

    Caron, M., Bojanowski, P., Joulin, A., Douze, M.: Deep clustering for unsuper- vised learning of visual features. In: Proceedings of the European Conference on Computer Vision (2018). https://doi.org/10.1007/978-3-030-01264-9_9

    work page doi:10.1007/978-3-030-01264-9_9 2018

  58. [60]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision (2019)

    Ji, X., Henriques, J.F., Vedaldi, A.: Invariant information clustering for unsuper- vised image classification and segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (2019). https://doi.org/10.1109/ iccv.2019.00996

    work page arXiv 2019

  59. [61]

    In: Advances in Neural Information Processing Systems, vol

    Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., Joulin, A.: Unsuper- vised learning of visual features by contrasting cluster assignments. In: Advances in Neural Information Processing Systems, vol. 33, pp. 9912–9924 (2020)

    work page 2020

  60. [62]

    In: Proceedings of the IEEE/CVF International ConferenceonComputerVision(2019).https://doi.org/10.1109/iccv.2019.00932

    Oh, S.W., Lee, J.-Y., Xu, N., Kim, S.J.: Video object segmentation using space-time memory networks. In: Proceedings of the IEEE/CVF International ConferenceonComputerVision(2019).https://doi.org/10.1109/iccv.2019.00932

    work page doi:10.1109/iccv.2019.00932 2019

  61. [63]

    In: Advances in Neural Information Processing Systems, vol

    Aydemir, G., Xie, W., Guney, F.: Self-supervised object-centric learning for videos. In: Advances in Neural Information Processing Systems, vol. 36, pp. 32879–32899 (2023). https://doi.org/10.52202/075280-1424

    work page doi:10.52202/075280-1424 2023

  62. [64]

    In: Advances in Neural Information Processing Systems, vol

    Xie, J., Xie, W., Zisserman, A.: Segmenting moving objects via an object-centric layered representation. In: Advances in Neural Information Processing Systems, vol. 35, pp. 28023–28036 (2022). https://doi.org/10.52202/068431-2032 33

    work page doi:10.52202/068431-2032 2022

  63. [65]

    Berg, Wan-Yen Lo, Piotr Doll ´ar, and Ross B

    Qian, R., Ding, S., Liu, X., Lin, D.: Semantics meets temporal correspon- dence: Self-supervised object-centric learning in videos. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (2023). https://doi. org/10.1109/iccv51070.2023.01529

    work page doi:10.1109/iccv51070.2023.01529 2023

  64. [66]

    IEEE Transactions on Multimedia (2020) https://doi.org/ 10.1109/tmm.2020.3026913

    Xu, K., Wen, L., Li, G., Huang, Q.: Self-supervised deep triplenet for video object segmentation. IEEE Transactions on Multimedia (2020) https://doi.org/ 10.1109/tmm.2020.3026913

    work page doi:10.1109/tmm.2020.3026913 2020

  65. [67]

    In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018)

    Yang, L., Wang, Y., Xiong, X., Yang, J., Katsaggelos, A.K.: Efficient video object segmentation via network modulation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018). https://doi.org/10.1109/ cvpr.2018.00680

    work page arXiv 2018

  66. [68]

    In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp

    Perazzi, F., Khoreva, A., Benenson, R., Schiele, B., Sorkine-Hornung, A.: Learn- ing video object segmentation from static images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2663–2672 (2017). https://doi.org/10.1109/cvpr.2017.372

    work page doi:10.1109/cvpr.2017.372 2017

  67. [69]

    In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018)

    Oh, S.W., Lee, J.-Y., Sunkavalli, K., Kim, S.J.: Fast video object segmentation by reference-guided mask propagation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018). https://doi.org/10.1109/ cvpr.2018.00770

    work page arXiv 2018

  68. [70]

    In: Conference on Computer Vision and Pattern Recognition

    Lu, X., Wang, W., Shen, J., Tai, Y.-W., Crandall, D.J., Hoi, S.C.: Learning video object segmentation from unlabeled videos. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2020). https://doi. org/10.1109/cvpr42600.2020.00898

    work page doi:10.1109/cvpr42600.2020.00898 2020

  69. [71]

    In: Proceedings of the AAAI Conference on Artificial Intelligence (2021)

    Wang, N., Zhou, W., Li, H.: Contrastive transformation for self-supervised cor- respondence learning. In: Proceedings of the AAAI Conference on Artificial Intelligence (2021). https://doi.org/10.1609/aaai.v35i11.17220

    work page doi:10.1609/aaai.v35i11.17220 2021

  70. [72]

    In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp

    Caelles,S.,Maninis,K.-K.,Pont-Tuset,J.,Leal-Taixé,L.,Cremers,D.,VanGool, L.: One-shot video object segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 221–230 (2017) 34

    work page 2017