Pith. sign in

REVIEW 3 major objections 5 minor 57 references

Self-Supervised Spatial Correspondence Across Modalities

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims dense cross-modal pixel correspondences can be learned from unlabeled video alone, with no photo-consistency assumptions or aligned multimodal pairs, by extending the contrastive random walk with both cross-modal and…

desk verdict A useful extension of GMRW to cross-modal dense matching, with strong results and useful new benchmarks, but the evaluation has rigor gaps and the cross-modal loss's grounding is not fully isolated. read the letter →

arxiv 2506.03148 v1 pith:IIKPSVEJ submitted 2025-06-03 cs.CV

Ayush Shrivastava , Andrew Owens This is my paper
classification cs.CV
keywords cross-modalcorrespondenceself-supervisedlearningcontrastiverandomwalkscycleconsistencyRGB-depthmatchingRGB-thermalsemanticpointtracking
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that dense, pixel-level correspondence between images from different visual modalities can be learned from unlabeled data alone. Given an RGB frame and a depth map or thermal image of the same scene taken at a different time and position, the model should identify the pixels that are the same physical point. The authors extend the contrastive random walk objective so that cycle consistency is enforced both between modalities and within each modality, and they show that this removes the need for spatially aligned multimodal training pairs, photo-consistency losses, or cross-modal translation. Reported results on RGB-depth and RGB-thermal benchmarks exceed those of optical-flow, translation-based, and diffusion-feature baselines, and on photo-sketch and cross-style matching the method is competitive with or better than specialized and supervised alternatives. If the claim holds, unlabeled multimodal video becomes a sufficient training signal for cross-modal registration and data fusion.

What carries the argument

The central machinery is the contrastive random walk on a cross-modal space-time graph, implemented with the global matching transformer architecture from [36]. Nodes are image patches from the two modalities; the transformer outputs correlation features whose scaled softmax inner product forms transition matrices $A^{m_1,m_2}_{t,t+k}$. Chaining the forward and backward transitions across a palindrome sequence and maximizing the return probability through the label-warping cross-entropy loss enforces cycle consistency, while auxiliary intra-modal random walks on augmented crops and an edge-aware smoothness loss stabilize the optimization. This machinery replaces hand-crafted photo-consistency with a learned, cycle-consistent notion of visual similarity that the model can share across modalities.

What would settle it

Train the identical pipeline on a modality pair whose signals share no occluding contours or edges, such as an RGB image matched against a randomly scrambled or heavily low-pass filtered version of depth, and measure the $\delta_x^{avg}$ accuracy against geometry-derived ground truth: if accuracy stays near chance, cycle consistency alone cannot resolve correspondences without shared geometric structure. A complementary check is to progressively blur or edge-erase one modality on the existing RGB-depth and RGB-thermal benchmarks and observe whether accuracy falls as shared contours are removed.

Watch

Extended reading notes

Core claim

The paper's central claim is that space-time cross-modal pixel correspondence can be learned purely from cycle consistency over unlabeled multimodal video. The model constructs a graph whose nodes are image patches in two modalities and whose edges carry random-walk transition probabilities computed by a shared global matching transformer. A palindrome sequence of the form modality A, modality B, modality A is used to require that a walker leaving a pixel in A, crossing into B, and returning to A lands where it started; maximizing that return probability via the label-warping objective trains the network. Because cross-modal intensity values are not comparable, the method adds intra-modal random walks between augmented crops of the same modality and an edge-aware smoothness loss to avoid degenerate solutions. The same training recipe, without aligned pairs or explicit similarity metrics, yields dense matches for RGB-depth, RGB-thermal, photo-sketch, and cross-style image pairs.

Load-bearing premise

The load-bearing premise is that the two modalities share enough visible geometric structure, such as edges and occluding contours, that cycle-consistent random walks latch onto physically correct matches; if a modality pair lacks such shared structure, the self-supervisory signal is ambiguous.

Editorial extensions

If this is right

  • Unlabeled multimodal video becomes a sufficient training signal: any corpus containing roughly overlapping RGB, depth, thermal, or other streams can be used to learn dense cross-modal correspondences without annotation or spatial alignment.
  • The absence of photo-consistency and translation assumptions means the same architecture and loss can be pointed at a new modality pair without redesign, as long as the pair shares visible structure such as occluding contours.
  • RGB-to-depth and RGB-to-thermal matching at the reported accuracies would make cross-modal registration, 3D reconstruction, and multimodal fusion pipelines work from loose, unsynchronized video collections instead of calibrated or aligned sensors.
  • For semantic matching, initializing the encoder with pretrained semantic features converts the cycle-consistency objective into a fine-grained alignment tool that is competitive with specialized photo-sketch systems and can match generated stylistic variants of the same scene.
  • The staged training recipe, intra-modal cycles first, then cross-modal cycles, then smoothness, is claimed to be necessary; skipping the intra-modal warm-up leaves the model in poor local optima with non-semantic matches.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An untested consequence of the paper's logic is that cycle consistency can be combined with other weak geometric cues, such as epipolar constraints, temporal flow, or partial calibration, to extend the method to modality pairs with little shared edge structure.
  • The paper's cross-style benchmark suggests a general recipe: generative text-to-image models can synthesize large paired training sets for semantic correspondence, which could be pushed to matching across lighting, season, or abstract renderings, not just art styles.
  • A testable prediction that follows from the stated limitation is that performance on a modality pair should track the amount of shared geometric structure; a pair like RGB and a heavily blurred or low-contrast thermal image should show proportionally lower accuracy.
  • If the method generalizes beyond visual sensors, the same random walk formulation might align modalities with event-boundary structure, such as audio spectrograms with video frames, as long as cycle consistency can find a common latent geometry.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a self-supervised method for dense cross-modal pixel correspondence, extending the Contrastive Random Walk (CRW)/GMRW framework to pairs of modalities such as RGB-depth, RGB-thermal, photo-sketch, and cross-style images. The model uses a shared transformer with per-modality encoders, computes transition matrices between all pairs of patches, and is trained with a cross-modal cycle-consistency loss, an intra-modal CRW loss on augmented views, and an edge-aware smoothness loss. Training requires only unlabeled multimodal videos/images, and evaluation covers geometric matching on NYU-Depth V2, Thermal-IM, and KAIST, as well as semantic matching on PSC6K and a newly constructed cross-style benchmark. The reported results show large gains over baselines on the geometric tasks and competitive or state-of-the-art numbers on the semantic tasks.

Significance. If the central claim holds, the paper would demonstrate that dense physical correspondences across rather different visual modalities can be learned without explicit photo-consistency assumptions, without spatially aligned multimodal pairs, and from unlabeled data. This is a meaningful extension of cycle-consistency-based correspondence learning and could broaden the applicability of self-supervised matching to multimodal sensor data. The paper is well structured, the method is described in enough detail to be reproduced from the appendix, and the ablations in Table 2 provide direct evidence for the value of intra-modal pretraining and the smoothness loss. The new cross-style benchmark and the careful description of the evaluation protocols are useful contributions. That said, several load-bearing points need additional support: the cycle-consistency objective is underdetermined unless some mechanism anchors the intermediate match to the physically correct point, and part of the geometric ground truth is generated by learned RGB trackers rather than independently measured.

major comments (3)
  1. [Section 3.2, Eq. (2)]
  2. [Section 4.1, RGB-Depth/RGB-Thermal Evaluation; Appendix A.2]
  3. [Abstract and Section 4.2]
minor comments (5)
  1. [Table 3]
  2. [Section 3.1, Eq. (1) and Eq. (4)]
  3. [Table 1]
  4. [Appendix A.2, Thermal-IM Evaluation]
  5. [Figure 2]

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the cross-modal cycle-consistency method is a new application of independently established prior work, and its evaluations use externally grounded labels rather than fitted inputs.

full rationale

The paper's central claim—that dense cross-modal correspondences can be learned from unlabeled multimodal videos—is not derived from its inputs by construction. Equation (2) is a cycle-consistency training loss inherited from CRW/GMRW; it enforces return-to-source under chained cross-modal transitions, but the paper does not define 'physical correspondence' as 'cycle-consistent match.' The possibility that Eq. (2) admits consistent but physically wrong intermediate assignments is an identifiability or correctness limitation, not a circular reduction: the method is validated against independently generated keypoints, including PIP++/CoTracker tracks checked by manual inspection, manual keypoint annotation for Thermal-IM and PSC6K, and manual annotation for the cross-style benchmark. The architecture and loss are taken from the authors' prior GMRW [36] and CRW [14], but these are published, externally evaluable methods with their own benchmarks; using them as a backbone is legitimate, and no parameter is fitted to the test labels or renamed as a prediction. The self-constructed benchmarks and the use of the same datasets for training and evaluation raise evaluation-validity questions, but they are not cases of a 'prediction' reducing to the training objective by definition. The paper's own limitation that all tested modalities share occluding contours is a generality limitation, not evidence of circularity. No uniqueness theorem, hidden ansatz, or renamed known result is load-bearing. Hence no significant circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim does not rely on fitted physical constants. The burdens are learning assumptions (cycle consistency, shared structure between modalities) and evaluation assumptions (pseudo-ground-truth labels, train/test separation), plus a small set of hand-set hyperparameters. No new physical entities are introduced.

free parameters (4)
  • Smoothness loss weight λ_s = 0 to 1, linearly increased over [100k, 120k] training steps (RGB-Depth); similar schedule elsewhere
    Hand-tuned; ablation (Table 2) shows it raises cross-modal accuracy substantially, e.g., RGB->D on NYU from 19.1 to 33.5.
  • Softmax temperature τ = sqrt(128)
    Adopted from GMFlow/GMRW, not tuned to the new tasks.
  • RandomResizedCrop size ratio = area (0.08, 1.0), aspect (0.7, 1.3)
    Copied from GMRW; controls the difficulty of intra-modal cycles.
  • Per-stage training iterations = RGB-Depth 50k/100k/20k; RGB-Thermal 30k/100k/20k; Photo-sketch 12k/10k/28k
    Chosen by hand; the paper does not describe a selection procedure, so these could partly encode task-specific tuning.
assumptions (4)
  • domain assumption Cycle-consistent random walks on a graph of patches produce geometrically meaningful cross-modal matches.
    The training objective (Eq. 2) maximizes return probability; the paper provides no proof that the learned transition probabilities correspond to physical points rather than to an arbitrary consistent relabeling. This is the core learning principle inherited from CRW/GMRW.
  • domain assumption The two modalities share enough visible structure (edges, occluding contours) to disambiguate the correct match.
    The Limitations section states that occluding contours visible in RGB, Thermal, and Depth are used for matching. If a modality pair lacks such shared structure, the cycle-consistency signal is ambiguous and the modality-agnostic claim fails.
  • domain assumption Pseudo ground-truth tracks (PIP++, CoTracker) and manual keypoints are accurate enough to measure correspondence quality.
    NYU-Depth evaluation uses PIP++ tracks propagated via calibration; Thermal-IM uses 100 manually annotated frame pairs. No analysis of label noise or inter-annotator agreement is provided.
  • domain assumption Evaluation clips are disjoint from training clips, or any overlap does not inflate reported accuracy.
    The paper trains on roughly 400K NYU frames and evaluates on 250 NYU clips but never states that these clips were excluded from training; the same issue applies to KAIST.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Self-Supervised Spatial Correspondence Across Modalities." pith.science (2026). https://pith.science/paper/IIKPSVEJ

@misc{pith2026250603148,
  author       = {Pith},
  title        = {Pith review of: Self-Supervised Spatial Correspondence Across Modalities},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IIKPSVEJ}},
  note         = {Machine review of arXiv:2506.03148}
}
read the original abstract

We present a method for finding cross-modal space-time correspondences. Given two images from different visual modalities, such as an RGB image and a depth map, our model identifies which pairs of pixels correspond to the same physical points in the scene. To solve this problem, we extend the contrastive random walk framework to simultaneously learn cycle-consistent feature representations for both cross-modal and intra-modal matching. The resulting model is simple and has no explicit photo-consistency assumptions. It can be trained entirely using unlabeled data, without the need for any spatially aligned multimodal image pairs. We evaluate our method on both geometric and semantic correspondence tasks. For geometric matching, we consider challenging tasks such as RGB-to-depth and RGB-to-thermal matching (and vice versa); for semantic matching, we evaluate on photo-sketch and cross-style image alignment. Our method achieves strong performance across all benchmarks.

Figures

Figures reproduced from arXiv: 2506.03148 by the authors.

Figure 1
Figure 1. Finding spatial correspondences across modalities. We present a method for cross-modal matching, trained entirely through self-supervision using a simple formulation based on contrastive random walks [14]. (a) Given two images taken by different visual modalities and at different positions and times, we predict the pairs of image patches that physically correspond to the same points. (b) We also apply our method to … view at source ↗
Figure 2
Figure 2. Model Architecture. We learn to find pixel-level correspondences between images that may differ in sensory modality, time, and scene position. Given images from two modalities (e.g., unpaired RGB and depth images from the same scene), we perform a contrastive random walk on a graph whose nodes come from patches within the two images using a global matching transformer architecture [36]. We simultaneously perform aux… view at source ↗
Figure 3
Figure 3. RGB-Depth Matching. We show qualitative compar￾isons on the NYU Depth V2 dataset. RAFT and GMFlow struggle to establish accurate correspondences in the depth domain, while our method successfully matches keypoints across RGB and depth images. The points shown are randomly sampled from the dataset annotations. ing its value as a regularization strategy. 5.2. Semantic correspondence Photo-Sketch Matching. In [PITH_FU… view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: RGB-Thermal Matching. Qualitative comparisons on the Thermal-IM (left) and KAIST (right) datasets. Our method accurately tracks keypoints across RGB and thermal images, even in the presence of motion. In contrast, RAFT and GMFlow often fail to produce meaningful corres…
Figure 5
Figure 5. Figure 5: Photo-Sketch Matching. Qualitative results of our method on the PSC6K dataset [28]. For each image pair, all annotated points from the dataset are used as query keypoints. prompt: a man in a diving suit and mask watercolor style anime style neon style light style pixel…
Figure 6
Figure 6. Figure 6: Cross-style Image Matching. We show qualitative results of our method on matching images across different styles. The images are generated using the Flux [19] model, with the prompts and styles used for generation shown above. methods on semantic matching. We see our w…
Figure 7
Figure 7. Figure 7: Qualitative examples for RGB-Depth matching with different losses. Zoom in for details. Points with red cross show the incorrect correspondences (not within 50 px distance of ground truth). A.1.3. Hyperparameters Here are the rest of the hyperparameters used in the mod…
Figure 8
Figure 8. Figure 8: CycleGAN+GMFlow Baseline. We show results for a CycleGAN-based baseline that performs image translation from RGB to the depth domain, followed by matching in the depth space. We also present the reverse setup, where images are translated from depth to RGB, and matching…
Figure 9
Figure 9. Figure 9: Cross-style Matching. We show qualitative results from our model on matching images generated in different styles by a image generation model. annotated tracks per clip. Thermal-IM Evaluation Datasets. In the Thermal-IM dataset, RGB and thermal images are not spatially…
Figure 10
Figure 10. Figure 10: RGB-Depth Matching. We show qualitative results from our model on NYU-Depth V2 dataset [37]. RGB to Thermal Thermal to RGB [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Thermal-IM Matching. We show qualitative results from our model on Thermal-IM [42] and KAIST datasets [13]. Photo-Sketch Matching [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]
Figure 12
Figure 12. Figure 12: Photo-Sketch Matching. We show qualitative results from our model on PSC6K dataset [28] [PITH_FULL_IMAGE:figures/full_fig_p015_12.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

57 extracted references · 45 canonical work pages

  1. [1]

    Unsupervised multi-modal image reg- istration via geometry preserving image-to-image translation

    Moab Arar, Yiftach Ginger, Dov Danon, Amit H Bermano, and Daniel Cohen-Or. Unsupervised multi-modal image reg- istration via geometry preserving image-to-image translation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020. 1, 2, 3, 5, 6, 7

    work page 2020

  2. [2]

    Learning pixel trajectories with multiscale contrastive random walks

    Zhangxing Bian, Allan Jabri, Alexei A Efros, and Andrew Owens. Learning pixel trajectories with multiscale contrastive random walks. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6508– 6519, 2022. 2, 4

    work page 2022

  3. [3]

    A simple framework for contrastive learning of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Ge- offrey Hinton. A simple framework for contrastive learning of visual representations. InInternational conference on machine learning, pages 1597–1607. PMLR, 2020. 2

    work page 2020

  4. [4]

    Tap-vid: A benchmark for tracking any point in a video.arXiv preprint arXiv:2211.03726, 2022

    Carl Doersch, Ankush Gupta, Larisa Markeeva, Adrià Re- casens, Lucas Smaira, Yusuf Aytar, João Carreira, Andrew Zisserman, and Yi Yang. Tap-vid: A benchmark for tracking any point in a video.arXiv preprint arXiv:2211.03726, 2022. 2, 3, 5, 13

    arXiv 2022

  5. [5]

    Tapir: Tracking any point with per-frame initialization and temporal refinement.arXiv preprint arXiv:2306.08637, 2023

    Carl Doersch, Yi Yang, Mel Vecerik, Dilara Gokay, Ankush Gupta, Yusuf Aytar, Joao Carreira, and Andrew Zisserman. Tapir: Tracking any point with per-frame initialization and temporal refinement.arXiv preprint arXiv:2306.08637, 2023. 3

    arXiv 2023

  6. [6]

    Tactile-augmented radiance fields.Computer Vision and Pattern Recognition (CVPR), 2024

    Yiming Dou, Fengyu Yang, Yi Liu, Antonio Loquercio, and Andrew Owens. Tactile-augmented radiance fields.Computer Vision and Pattern Recognition (CVPR), 2024. 1

    work page 2024

  7. [7]

    Watching the world go by: Representation learning from unlabeled videos.arXiv preprint arXiv:2003.07990, 2020

    Daniel Gordon, Kiana Ehsani, Dieter Fox, and Ali Farhadi. Watching the world go by: Representation learning from unlabeled videos.arXiv preprint arXiv:2003.07990, 2020. 3

    arXiv 2003

  8. [8]

    Kubric: A scalable dataset generator

    Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David J Fleet, Dan Gnanapra- gasam, Florian Golemo, Charles Herrmann, et al. Kubric: A scalable dataset generator. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3749–3761, 2022. 3

    work page 2022

Show all 57 references
  1. [9]

    Particle video revisited: Tracking through occlusions using point trajectories

    Adam W Harley, Zhaoyuan Fang, and Katerina Fragkiadaki. Particle video revisited: Tracking through occlusions using point trajectories. InEuropean Conference on Computer Vision, pages 59–75. Springer, 2022. 3

    2022

  2. [10]

    Momentum contrast for unsupervised visual rep- resentation learning

    Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual rep- resentation learning. InProceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pages 9729–9738, 2020. 2

    2020

  3. [11]

    Mix and localize: Localizing sound sources in mixtures.Computer Vision and Pattern Recognition (CVPR), 2022

    Xixi Hu, Ziyang Chen, and Andrew Owens. Mix and localize: Localizing sound sources in mixtures.Computer Vision and Pattern Recognition (CVPR), 2022. 2

    2022

  4. [12]

    Dynamic context correspondence network for semantic alignment

    Shuaiyi Huang, Qiuyue Wang, Songyang Zhang, Shipeng Yan, and Xuming He. Dynamic context correspondence network for semantic alignment. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 2010–2019, 2019. 7

    2010

  5. [13]

    Multispectral pedestrian detection: Benchmark dataset and baselines

    Soonmin Hwang, Jaesik Park, Namil Kim, Yukyung Choi, and In So Kweon. Multispectral pedestrian detection: Benchmark dataset and baselines. InProceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015. 4, 15

    2015

  6. [14]

    Space-time correspondence as a contrastive random walk.Advances in neural information processing systems, 33:19545–19560,

    Allan Jabri, Andrew Owens, and Alexei Efros. Space-time correspondence as a contrastive random walk.Advances in neural information processing systems, 33:19545–19560,

  7. [15]

    Spatial transformer networks.Advances in neural information processing systems, 28, 2015

    Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. Spatial transformer networks.Advances in neural information processing systems, 28, 2015. 2

    2015

  8. [16]

    Unsupervised learning of multi-frame optical flow with occlusions

    Joel Janai, Fatma Guney, Anurag Ranjan, Michael Black, and Andreas Geiger. Unsupervised learning of multi-frame optical flow with occlusions. InProceedings of the European conference on computer vision (ECCV), pages 690–706, 2018. 2, 3

    2018

  9. [17]

    What matters in unsupervised optical flow

    Rico Jonschkowski, Austin Stone, Jonathan T Barron, Ariel Gordon, Kurt Konolige, and Anelia Angelova. What matters in unsupervised optical flow. InComputer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23– 28, 2020, Proceedings, Part II 16, pages 557–572. Springer,

    2020

  10. [18]

    Co- tracker: It is better to track together

    Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Co- tracker: It is better to track together. InProc. ECCV, 2024. 5, 14

    2024

  11. [19]

    Black Forest Labs. Flux. https://github.com/ black-forest-labs/flux, 2024. 1, 2, 5, 8

    2024

  12. [20]

    Mast: A memory- augmented self-supervised tracker

    Zihang Lai, Erika Lu, and Weidi Xie. Mast: A memory- augmented self-supervised tracker. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6479–6488, 2020. 3

    2020

  13. [21]

    Blip- 2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip- 2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InInternational conference on machine learning, pages 19730–19742. PMLR,

  14. [22]

    2d3d-matr: 2d-3d matching transformer for detection-free registration between images and point clouds

    Minhao Li, Zheng Qin, Zhirui Gao, Renjiao Yi, Chenyang Zhu, Yulan Guo, and Kai Xu. 2d3d-matr: 2d-3d matching transformer for detection-free registration between images and point clouds. InProceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 14128–14138,

  15. [23]

    Joint-task self-supervised learning for temporal correspondence.Advances in Neural Information Processing Systems, 32, 2019

    Xueting Li, Sifei Liu, Shalini De Mello, Xiaolong Wang, Jan Kautz, and Ming-Hsuan Yang. Joint-task self-supervised learning for temporal correspondence.Advances in Neural Information Processing Systems, 32, 2019. 2, 3

    2019

  16. [24]

    Probabilistic model distillation for semantic correspondence

    Xin Li, Deng-Ping Fan, Fan Yang, Ao Luo, Hong Cheng, and Zicheng Liu. Probabilistic model distillation for semantic correspondence. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7505– 7514, 2021. 7

    2021

  17. [25]

    Learning by analogy: Reliable supervision from transformations for unsupervised optical flow estima- tion

    Liang Liu, Jiangning Zhang, Ruifei He, Yong Liu, Yabiao Wang, Ying Tai, Donghao Luo, Chengjie Wang, Jilin Li, and Feiyue Huang. Learning by analogy: Reliable supervision from transformations for unsupervised optical flow estima- tion. InIEEE Conference on Computer Vision and P...

    2020

  18. [26]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021. 12

    2021

  19. [27]

    Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017. 12

    2017 arXiv

  20. [28]

    Learn- ing dense correspondences between photos and sketches

    Xuanchen Lu, Xiaolong Wang, and Judith E Fan. Learn- ing dense correspondences between photos and sketches. In International Conference on Machine Learning, pages 22899– 22916. PMLR, 2023. 1, 3, 4, 5, 6, 7, 8, 15

    2023

  21. [29]

    Dinov2: Learning robust visual features without supervision.arXiv preprint arXiv:2304.07193, 2023

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision.arXiv preprint arXiv:2304.07193, 2023. 1, 2, 4, 6, 7

    2023 arXiv

  22. [30]

    Pytorch: An imperative style, high-performance deep learning library.Ad- vances in neural information processing systems, 32, 2019

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library.Ad- vances in neural information processing systems, 32...

    2019

  23. [31]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning. PMLR,

  24. [32]

    E. Riba, D. Mishkin, D. Ponsa, E. Rublee, and G. Bradski. Ko- rnia: an open source differentiable computer vision library for pytorch. InWinter Conference on Applications of Computer Vision, 2020. 12

    2020

  25. [33]

    End-to- end weakly-supervised semantic alignment

    Ignacio Rocco, Relja Arandjelovi´c, and Josef Sivic. End-to- end weakly-supervised semantic alignment. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6917–6925, 2018. 7

    2018

  26. [34]

    Neighbourhood con- sensus networks.Advances in neural information processing systems, 31, 2018

    Ignacio Rocco, Mircea Cimpoi, Relja Arandjelovi´c, Akihiko Torii, Tomas Pajdla, and Josef Sivic. Neighbourhood con- sensus networks.Advances in neural information processing systems, 31, 2018. 7

    2018

  27. [35]

    Imagenet large scale visual recognition challenge.International journal of computer vision, 115:211–252, 2015

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, San- jeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge.International journal of computer vision, 115:211–252, 2015. 5

    2015

  28. [36]

    Self-supervised any- point tracking by contrastive random walks

    Ayush Shrivastava and Andrew Owens. Self-supervised any- point tracking by contrastive random walks. InEuropean Conference on Computer Vision (ECCV), 2024. 2, 3, 4, 5, 6, 7, 12

    2024

  29. [37]

    Indoor segmentation and support inference from rgbd images

    Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. InComputer Vision–ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7- 13, 2012, Proceedings, Part V 12, pages 746–760. Springer,

    2012

  30. [38]

    Smurf: Self-teaching multi-frame unsupervised raft with full-image warping

    Austin Stone, Daniel Maurer, Alper Ayvaci, Anelia Angelova, and Rico Jonschkowski. Smurf: Self-teaching multi-frame unsupervised raft with full-image warping. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3887–3896, 2021. 2, 3

    2021

  31. [39]

    Autoflow: Learning a better training set for optical flow

    Deqing Sun, Daniel Vlasic, Charles Herrmann, Varun Jampani, Michael Krainin, Huiwen Chang, Ramin Zabih, William T Freeman, and Ce Liu. Autoflow: Learning a better training set for optical flow. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition...

    2021

  32. [40]

    Emergent correspondence from image diffusion

    Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. Emergent correspondence from image diffusion. InThirty-seventh Conference on Neural Information Processing Systems, 2023. 5, 6, 7

    2023

  33. [41]

    Breaking shortcut: Ex- ploring fully convolutional cycle-consistency for video corre- spondence learning.arXiv preprint arXiv:2105.05838, 2021

    Yansong Tang, Zhenyu Jiang, Zhenda Xie, Yue Cao, Zheng Zhang, Philip HS Torr, and Han Hu. Breaking shortcut: Ex- ploring fully convolutional cycle-consistency for video corre- spondence learning.arXiv preprint arXiv:2105.05838, 2021. 2, 4

    2021 arXiv

  34. [42]

    What happened 3 seconds ago? inferring the past with thermal imag- ing

    Zitian Tang, Wenjie Ye, Wei-Chiu Ma, and Hang Zhao. What happened 3 seconds ago? inferring the past with thermal imag- ing. InProceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 17111–17120,

  35. [43]

    Raft: Recurrent all-pairs field transforms for optical flow

    Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. InComputer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pages 402–419. Springer, 2020. 3, 5, 6

    2020

  36. [44]

    Contrastive multiview coding

    Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. InComputer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XI 16, 2020. 2

    2020

  37. [45]

    Warp consistency for unsupervised learning of dense correspondences

    Prune Truong, Martin Danelljan, Fisher Yu, and Luc Van Gool. Warp consistency for unsupervised learning of dense correspondences. InProceedings of the IEEE/CVF international conference on computer vision, pages 10346– 10356, 2021. 7

    2021

  38. [46]

    Tracking emerges by col- orizing videos

    Carl V ondrick, Abhinav Shrivastava, Alireza Fathi, Sergio Guadarrama, and Kevin Murphy. Tracking emerges by col- orizing videos. InProceedings of the European conference on computer vision (ECCV), pages 391–408, 2018. 3

    2018

  39. [47]

    Freereg: Image- to-point cloud registration leveraging pretrained diffusion models and monocular depth estimators.arXiv preprint arXiv:2310.03420, 2023

    Haiping Wang, Yuan Liu, Bing Wang, Yujing Sun, Zhen Dong, Wenping Wang, and Bisheng Yang. Freereg: Image- to-point cloud registration leveraging pretrained diffusion models and monocular depth estimators.arXiv preprint arXiv:2310.03420, 2023. 1, 3

    arXiv

  40. [48]

    Learning correspondence from the cycle-consistency of time

    Xiaolong Wang, Allan Jabri, and Alexei A Efros. Learning correspondence from the cycle-consistency of time. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2566–2576, 2019. 2

    2019

  41. [49]

    Gmflow: Learning optical flow via global matching

    Haofei Xu, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, and Dacheng Tao. Gmflow: Learning optical flow via global matching. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8121–8130,

  42. [50]

    Rethinking self-supervised correspondence learning: A video frame-level similarity per- spective

    Jiarui Xu and Xiaolong Wang. Rethinking self-supervised correspondence learning: A video frame-level similarity per- spective. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 10075–10085, 2021. 3

    2021

  43. [51]

    Yu, Adam W

    Jason J. Yu, Adam W. Harley, and Konstantinos G. Derpanis. Back to basics: Unsupervised learning of optical flow via brightness constancy and motion smoothness. InComputer Vision - ECCV 2016 Workshops, Part 3, 2016. 2, 3

    2016

  44. [52]

    A Tale of Two Features: Stable Diffusion Comple- ments DINO for Zero-Shot Semantic Correspondence.arXiv preprint arxiv:2305.15347, 2023

    Junyi Zhang, Charles Herrmann, Junhwa Hur, Luisa Pola- nia Cabrera, Varun Jampani, Deqing Sun, and Ming-Hsuan Yang. A Tale of Two Features: Stable Diffusion Comple- ments DINO for Zero-Shot Semantic Correspondence.arXiv preprint arxiv:2305.15347, 2023. 5, 6, 7

    2023 arXiv

  45. [53]

    Telling left from right: Identifying geometry-aware semantic cor- respondence

    Junyi Zhang, Charles Herrmann, Junhwa Hur, Eric Chen, Varun Jampani, Deqing Sun, and Ming-Hsuan Yang. Telling left from right: Identifying geometry-aware semantic cor- respondence. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), page...

    2024

  46. [54]

    Harley, Bokui Shen, Gordon Wet- zstein, and Leonidas J

    Yang Zheng, Adam W. Harley, Bokui Shen, Gordon Wet- zstein, and Leonidas J. Guibas. Pointodyssey: A large-scale synthetic dataset for long-term point tracking. InICCV, 2023. 5, 13

    2023

  47. [55]

    Learning dense correspondence via 3d-guided cycle consistency

    Tinghui Zhou, Philipp Krahenbuhl, Mathieu Aubry, Qixing Huang, and Alexei A Efros. Learning dense correspondence via 3d-guided cycle consistency. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 117–126, 2016. 2

    2016

  48. [56]

    Unpaired image-to-image translation using cycle-consistent adversarial networks

    Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. InComputer Vision (ICCV), 2017 IEEE International Conference on, 2017. 6

    2017

  49. [57]

    dcpQpF0qlGJsE4HlBypEHB63uE0=

    Yuliang Zou, Zelun Luo, and Jia-Bin Huang. Df-net: Un- supervised joint learning of depth and flow using cross-task consistency. InProceedings of the European conference on computer vision (ECCV), pages 36–53, 2018. 2, 3 A.1. Implementation details Here, we present the model a...

    2018

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.