REVIEW 3 major objections 5 minor 2 cited by
Emergent Temporal Correspondences from Video Diffusion Transformers
T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read Temporal correspondences in video diffusion transformers emerge from query-key similarity in a few dominant attention layers, and can be extracted for zero-shot point tracking and motion-enhanced generation.
desk verdict A useful empirical map of where temporal correspondence lives in video DiTs, with a real zero-shot tracking payoff—but the layer rankings are selected against CoTracker pseudo-labels, so the mechanism claims are suggestive, not settled. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is full 3D attention in DiT blocks, where each layer computes softmax($QK^\top/\sqrt{d}$) over a sequence containing all frame latents and text tokens. The relevant sub-block is cross-frame attention $A^{i,j}_{t,l}$, the attention from tokens of frame $i$ to tokens of frame $j$, whose per-pixel argmax gives a predicted match. DiffTrack's machinery is the query-key matching cost $C^{1,j}_{t,l} = \mathrm{Softmax}(D^1_{t,l}(D^j_{t,l})^\top/\sqrt{d})$ with descriptors $D$ set to queries and keys rather than intermediate features, together with three metrics: matching accuracy (PCK against pseudo ground truth), confidence (maximum attention value), and attention score (sum of cross-frame attention). Their harmonic mean identifies the dominant layer and timestep, e.g. $l=17$, $t=1$ for CogVideoX-2B. The same cross-frame attention maps are the target of Cross-Attention Guidance, which zeros them out in chosen layers to create a degraded motion sample and guides the model away from it.
What would settle it
Manually annotate point trajectories on a sample of the generated videos, or replace CoTracker with an independent tracker, and rerun the layer and timestep selection; if the top descriptors move away from $l=17$, $t=1$ for CogVideoX-2B or the layer ranking changes substantially, the claimed locus of temporal correspondence is an artifact of the pseudo-labeling.
Extended reading notes
Core claim
The central discovery is that video DiTs establish temporal correspondences through query-key matching in full 3D attention: the softmax query-key product between latents of different frames acts as a matching cost that tracks corresponding points across frames. This matching is not uniform across the network; a handful of layers are responsible, and for CogVideoX-2B the single best descriptor location is layer 17 at the final denoising timestep $t=1$. Matching accuracy is highest late in denoising, with a slight degradation at the very end. The same pattern holds across CogVideoX-5B and HunyuanVideo, and an analysis on real videos using noised latents reproduces the layer and timestep conclusions. DiffTrack treats these attention-derived descriptors as a zero-shot tracker, achieving the best average point accuracy on TAP-Vid among the compared image foundation models and self-supervised video models, and it uses the identified dominant layers to design Cross-Attention Guidance, which improves motion consistency.
Load-bearing premise
The analysis depends on the assumption that CoTracker's estimated trajectories, used as stand-in ground truth for videos the model itself generated, are accurate enough to rank layers and timesteps; if CoTracker systematically misses or biases certain motions, the identified dominant layer could be tuned to CoTracker rather than to true physical correspondence.
Editorial extensions
If this is right
- Zero-shot point tracking: query-key descriptors from one selected layer and timestep outperform single-image foundation models and self-supervised video models on TAP-Vid average accuracy.
- Feature selection matters: using the single dominant layer and timestep is much better than averaging over all layers or timesteps, and fusing multiple timesteps and layers does not improve temporal matching.
- Cross-frame attention is causally involved in motion: zeroing it in the dominant layers and guiding away from the degraded sample improves subject consistency, background consistency, dynamic degree, and imaging quality without training.
- The pattern generalizes across architectures: CogVideoX-5B and HunyuanVideo show the same query-key dominance, concentration in a few layers, and strengthening during denoising, while the image-to-video variant loses temporal matching as self-frame attention dominates.
Reading between the lines
- An independent test with human-labeled trajectories on a sample of the generated videos would tell whether the layer and timestep conclusions describe physical motion or CoTracker's bias; this is testable with relatively small annotation effort.
- If the layer specialization is robust, practitioners could speed up video DiTs by restricting full cross-frame attention to the few dominant layers for motion-relevant steps, or reuse those layers as a training-free motion prior for tracking, flow-like estimation, and video editing.
- The late-denoising sharpening suggests motion is committed near the end of sampling, which may explain why trajectory-conditioned control methods are effective and why early-step motion conditioning is insufficient.
- The guidance result implies cross-frame attention is a source of temporal consistency rather than a purely descriptive byproduct; this predicts that selectively degrading cross-frame attention in dominant layers during sampling will produce characteristic motion errors such as frozen or duplicated frames.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces DiffTrack, a framework for quantitatively analyzing how video diffusion transformers (DiTs) establish temporal correspondences during generation. The authors construct a dataset of prompt-generated videos with pseudo ground-truth trajectories from CoTracker, propose three metrics (matching accuracy, confidence score, and attention score) combined via a harmonic mean, and use these to analyze representation type (query-key vs. intermediate features), layer, and denoising timestep in CogVideoX-2B, CogVideoX-5B, HunyuanVideo, and CogVideoX-2B-I2V. They report that query-key similarities in a few specific layers dominate temporal matching and that matching strengthens as denoising progresses. The findings are applied to zero-shot point tracking on TAP-Vid, where the approach achieves high average accuracy, and to Cross-Attention Guidance (CAG), a training-free guidance method that improves motion consistency in generated videos.
Significance. If the findings hold, the work is a valuable step toward understanding where and when video DiTs encode motion information, and it demonstrates practical payoffs in zero-shot tracking and motion-enhanced generation. The paper is commendably concrete: it evaluates multiple open-source DiT backbones, provides ablations for the proposed tracking architecture (Tables A.2-A.4), and validates CAG with both automatic metrics and human evaluation. The external TAP-Vid benchmark provides a strong check on the practical utility of the selected layer and timestep. However, the central scientific claim about which layers and timesteps matter is currently tied to a single pseudo-label source, so the analysis needs independent validation before the conclusions can be regarded as robust.
major comments (3)
- [Sec. 3.1 and Sec. 3.3] The layer-wise and timestep rankings that drive the paper's central claims are computed against pseudo ground-truth tracks produced by CoTracker. The matching accuracy metric (Eqs. 5-7) is evaluated against T from CoTracker, and the confidence and attention scores are averaged only over points CoTracker marks visible. Consequently, the selection of l=17, t=1, and the CAG layers l=13,17,21 is a measurement of agreement with CoTracker's inductive biases rather than with ground-truth physical motion. The real-video analysis in Fig. A.6 also uses CoTracker as the reference, so it does not break this dependency. Because the paper's main scientific claim is about where temporal correspondences are established inside video DiTs, this is load-bearing. I recommend validating the rankings with an independent reference, such as human-annotated TAP-Vid tracks, a second point tracker, or synthetic sequences with known ground-truth motion, and showing that the layer/timestep selections are stable.
- [Abstract and Sec. 3.4 vs. Fig. A.4(c)] The abstract and Section 3.4 state that temporal matching becomes increasingly prominent during the denoising process, but the authors' own analysis of CogVideoX-2B-I2V (Fig. A.4(c)) shows a sharp decline at later timesteps, and Table A.1 shows a corresponding drop in point accuracy. The claim in the abstract is therefore too strong and should be qualified as model-dependent, or the exception should be discussed in the main text.
- [Sec. 4, Table 1] The TAP-Vid benchmark validates the zero-shot tracking application but does not independently verify the internal analysis: the layer and timestep were selected using CoTracker-based pseudo-GT before Table 1 was computed. The paper states that the benchmark results 'highlight the in-depth analysis,' but the benchmark cannot serve as evidence for the rankings themselves. Please either provide an independent validation of the internal rankings or explicitly state that the benchmark only confirms the practical quality of the selected features.
minor comments (5)
- [Sec. 3.3] The normalization scheme used before computing the harmonic mean of matching accuracy, confidence score, and attention score is not described; please specify how each metric is normalized across layers and timesteps.
- [Sec. 4.1 and Table 1] For the fair-evaluation comparison, most baselines are resized to produce 30x45 feature maps, while ZeroCo uses a 14x14 feature map at 224x224 input; the potential effect of this resolution mismatch on the comparison should be discussed.
- [Fig. 4] The layer-wise analysis in Fig. 4(b) reports top-20/top-50 scores without error bars or statistical significance across the 50 prompts per dataset; adding variance or a significance test would strengthen the claim that a specific layer dominates.
- [Sec. C.2] There is a typo in the implementation details: 'for for CogVideoX-5B' should read 'for CogVideoX-5B'.
- [Eq. (9)] The notation for the set of cross-frame indices F and the number of frames F in the preliminaries is used interchangeably; please disambiguate the symbol.
Circularity Check
No circular derivation: layer/timestep selection is measured against CoTracker pseudo-GT on generated videos and then externally validated on human-annotated TAP-Vid; self-citations are interpretive, not load-bearing.
full rationale
DiffTrack's derivation chain is not circular. The central analysis in Sec. 3.4 chooses representations, layers, and timesteps by matching accuracy, confidence, and attention scores computed against CoTracker pseudo-GT on prompt-generated videos (Sec. 3.1: 'we use an off-the-shelf tracking method, CoTracker [42], to obtain pseudo ground truth T'). The selected configuration (l=17, t=1 for CogVideoX-2B) is then applied to real-video zero-shot point tracking and evaluated on human-annotated TAP-Vid (Sec. 4.2, Table 1). No TAP-Vid accuracy value is used to select the layer or timestep, so the TAP-Vid result is an external generalization check rather than a fitted prediction. The query-key versus intermediate-feature comparison is measured directly from the DiT attention machinery (Eqs. 5-6, 8-9), not imported from prior work; citations to the authors' earlier correspondence papers [3,54] appear only as supporting interpretation ('This finding aligns with prior works [3,54]') after the paper's own Fig. 4(a) provides the measured comparison. Likewise, CAG's borrowings from PAG [2] are implementation choices, and CAG is evaluated on VBench metrics that are independent of the attention analysis that selects the guided layers. The use of CoTracker pseudo-GT is a genuine validity concern, because the layer/timestep rankings could reflect CoTracker's inductive biases rather than true physical correspondence, but that is not circularity: CoTracker is an external reference and the paper's conclusions are not defined so as to reproduce CoTracker's outputs by construction. The minor overlap between the present authors and the cited correspondence works is not load-bearing, so the appropriate finding is a low non-circularity score.
Assumptions & free parameters
free parameters (3)
- Guidance scale s for Cross-Attention Guidance =
not reported
- Number of top layers for CAG =
3 (e.g., l=13,17,21 for CogVideoX-2B)
- Number of frames per chunk in zero-shot tracking =
13 (selected from ablation in Fig. A.12)
assumptions (5)
- domain assumption CoTracker's pseudo-labels are a valid proxy for true temporal correspondence in generated videos.
- domain assumption The curated prompt-generated videos are representative of the video DiT's behavior.
- domain assumption Query-key similarity in attention encodes geometric correspondence better than intermediate features.
- ad hoc to paper The harmonic mean of normalized matching accuracy, confidence, and attention score identifies the layers that matter for temporal correspondence.
- domain assumption DDIM inversion error is negligible at the final timestep t=1 for real videos.
Cite this review
Pith. "Pith review of Emergent Temporal Correspondences from Video Diffusion Transformers." pith.science (2026). https://pith.science/paper/L5YFUBGB
@misc{pith2026250617220,
author = {Pith},
title = {Pith review of: Emergent Temporal Correspondences from Video Diffusion Transformers},
year = {2026},
howpublished = {\url{https://pith.science/paper/L5YFUBGB}},
note = {Machine review of arXiv:2506.17220}
}
read the original abstract
Recent advancements in video diffusion models based on Diffusion Transformers (DiTs) have achieved remarkable success in generating temporally coherent videos. Yet, a fundamental question persists: how do these models internally establish and represent temporal correspondences across frames? We introduce DiffTrack, the first quantitative analysis framework designed to answer this question. DiffTrack constructs a dataset of prompt-generated video with pseudo ground-truth tracking annotations and proposes novel evaluation metrics to systematically analyze how each component within the full 3D attention mechanism of DiTs (e.g., representations, layers, and timesteps) contributes to establishing temporal correspondences. Our analysis reveals that query-key similarities in specific, but not all, layers play a critical role in temporal matching, and that this matching becomes increasingly prominent during the denoising process. We demonstrate practical applications of DiffTrack in zero-shot point tracking, where it achieves state-of-the-art performance compared to existing vision foundation and self-supervised video models. Further, we extend our findings to motion-enhanced video generation with a novel guidance method that improves temporal consistency of generated videos without additional training. We believe our work offers crucial insights into the inner workings of video DiTs and establishes a foundation for further research and applications leveraging their temporal understanding.
Figures
Figures from the paper (6 more)
Forward citations
Cited by 2 Pith papers
-
ASTRA: Asynchronous Spatio-Temporal Reconstruction via Trajectory Alignment
ASTRA jointly estimates camera time offsets and dynamic Gaussian geometry by aligning projected 3D motion with observed 2D trajectory tracks, improving robustness to large asynchrony.
-
Controlling Motion Transfer in Diffusion Transformers via Attention Heads
Video DiTs encode motion and structure in separate attention-head subsets; selecting and guiding those heads yields training-free motion transfer with higher fidelity and structural alignment than existing methods.
Reference graph
Works this paper leans on
-
[1]
GPT-4 technical report.arXiv preprint arXiv:2303.08774, 2023
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report.arXiv preprint arXiv:2303.08774, 2023
arXiv 2023
-
[2]
Self-rectifying diffusion sampling with perturbed-attention guidance
Donghoon Ahn, Hyoungwon Cho, Jaewon Min, Wooseok Jang, Jungwoo Kim, SeonHwa Kim, Hyun Hee Park, Kyong Hwan Jin, and Seungryong Kim. Self-rectifying diffusion sampling with perturbed-attention guidance. InECCV, pages 1–17, 2024
2024
-
[3]
Honggyu An, Jinhyeon Kim, Seonghoon Park, Jaewoo Jung, Jisang Han, Sunghwan Hong, and Seun- gryong Kim. Cross-view completion models are zero-shot correspondence estimators.arXiv preprint arXiv:2412.09072, 2024
arXiv 2024
-
[4]
Jie An, Songyang Zhang, Harry Yang, Sonal Gupta, Jia-Bin Huang, Jiebo Luo, and Xi Yin. Latent-shift: Latent diffusion with temporal shift for efficient text-to-video generation.arXiv preprint arXiv:2304.08477, 2023
arXiv 2023
-
[5]
Can visual foundation models achieve long-term point tracking?arXiv preprint arXiv:2408.13575, 2024
Görkay Aydemir, Weidi Xie, and Fatma Güney. Can visual foundation models achieve long-term point tracking?arXiv preprint arXiv:2408.13575, 2024
arXiv 2024
-
[6]
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127, 2023
arXiv 2023
-
[7]
Align your latents: High-resolution video synthesis with latent diffusion models
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. InCVPR, pages 22563–22575, 2023
2023
-
[8]
Minghong Cai, Xiaodong Cun, Xiaoyu Li, Wenze Liu, Zhaoyang Zhang, Yong Zhang, Ying Shan, and Xiangyu Yue. DiTCtrl: Exploring attention control in multi-modal diffusion transformer for tuning-free multi-prompt longer video generation.arXiv preprint arXiv:2412.18597, 2024
arXiv 2024
Show all 83 references
-
[9]
Zhang, Philipp Henzler, Zhengqi Li, Noah Snavely, and Ricardo Martin-Brualla
Ruojin Cai, Jason Y . Zhang, Philipp Henzler, Zhengqi Li, Noah Snavely, and Ricardo Martin-Brualla. Can generative video models help pose estimation?arXiv preprint arXiv:2412.16155, 2024
2024 arXiv
-
[10]
Emerging properties in self-supervised vision transformers
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. InICCV, pages 9650–9660, 2021
2021
-
[11]
VideoJAM: Joint appearance-motion representations for enhanced motion generation in video models.arXiv preprint arXiv:2502.02492, 2025
Hila Chefer, Uriel Singer, Amit Zohar, Yuval Kirstain, Adam Polyak, Yaniv Taigman, Lior Wolf, and Shelly Sheynin. VideoJAM: Joint appearance-motion representations for enhanced motion generation in video models.arXiv preprint arXiv:2502.02492, 2025
2025 arXiv
-
[12]
Videocrafter1: Open diffusion models for high-quality video generation.arXiv preprint arXiv:2310.19512, 2023
Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al. Videocrafter1: Open diffusion models for high-quality video generation.arXiv preprint arXiv:2310.19512, 2023
-
[13]
CATs: Cost aggregation transformers for visual correspondence.NeurIPS, 34:9011–9023, 2021
Seokju Cho, Sunghwan Hong, Sangryul Jeon, Yunsung Lee, Kwanghoon Sohn, and Seungryong Kim. CATs: Cost aggregation transformers for visual correspondence.NeurIPS, 34:9011–9023, 2021
2021
-
[14]
CATs++: Boosting cost aggregation with convolutions and transformers.IEEE TPAMI, 45(6):7174–7194, 2022
Seokju Cho, Sunghwan Hong, and Seungryong Kim. CATs++: Boosting cost aggregation with convolutions and transformers.IEEE TPAMI, 45(6):7174–7194, 2022
2022
-
[15]
Seurat: From moving points to depth
Seokju Cho, Jiahui Huang, Seungryong Kim, and Joon-Young Lee. Seurat: From moving points to depth. arXiv preprint arXiv:2504.14687, 2025
2025 arXiv
-
[16]
Local all-pair correspondence for point tracking
Seokju Cho, Jiahui Huang, Jisu Nam, Honggyu An, Seungryong Kim, and Joon-Young Lee. Local all-pair correspondence for point tracking. InECCV, pages 306–325, 2024
2024
-
[17]
Vision transformers need registers
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. arXiv preprint arXiv:2309.16588, 2023
2023 arXiv
-
[18]
TAP-Vid: A benchmark for tracking any point in a video.NeurIPS, 35:13610–13626, 2022
Carl Doersch, Ankush Gupta, Larisa Markeeva, Adria Recasens, Lucas Smaira, Yusuf Aytar, Joao Carreira, Andrew Zisserman, and Yi Yang. TAP-Vid: A benchmark for tracking any point in a video.NeurIPS, 35:13610–13626, 2022
2022
-
[19]
TAPIR: Tracking any point with per-frame initialization and temporal refinement
Carl Doersch, Yi Yang, Mel Vecerik, Dilara Gokay, Ankush Gupta, Yusuf Aytar, Joao Carreira, and Andrew Zisserman. TAPIR: Tracking any point with per-frame initialization and temporal refinement. InICCV, pages 10061–10072, 2023. A.22
2023
-
[20]
Scaling rectified flow transformers for high-resolution image synthesis
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. InICML, 2024
2024
-
[21]
Perceptual quality assessment of smartphone photography
Yuming Fang, Hanwei Zhu, Yan Zeng, Kede Ma, and Zhou Wang. Perceptual quality assessment of smartphone photography. InCVPR, pages 3677–3686, 2020
2020
-
[22]
Barron, and Ben Poole
Ruiqi Gao, Aleksander Holynski, Philipp Henzler, Arthur Brussee, Ricardo Martin-Brualla, Pratul Srini- vasan, Jonathan T. Barron, and Ben Poole. CAT3D: Create anything in 3D with multi-view diffusion models.arXiv preprint arXiv:2405.10314, 2024
2024 arXiv
-
[23]
Motion prompting: Controlling video generation with motion trajectories.arXiv preprint arXiv:2412.02700, 2024
Daniel Geng, Charles Herrmann, Junhwa Hur, Forrester Cole, Serena Zhang, Tobias Pfaff, Tatiana Lopez- Guevara, Carl Doersch, Yusuf Aytar, Michael Rubinstein, et al. Motion prompting: Controlling video generation with motion trajectories.arXiv preprint arXiv:2412.02700, 2024
2024 arXiv
-
[24]
Mochi 1: A new SOTA in open-source video generation models, 2023
Genmo. Mochi 1: A new SOTA in open-source video generation models, 2023
2023
-
[25]
SparseCtrl: Adding sparse controls to text-to-video diffusion models
Yuwei Guo, Ceyuan Yang, Anyi Rao, Maneesh Agrawala, Dahua Lin, and Bo Dai. SparseCtrl: Adding sparse controls to text-to-video diffusion models. InECCV, pages 330–348, 2024
2024
-
[26]
AnimateDiff: Animate your personalized text-to-image diffusion models without specific tuning.arXiv preprint arXiv:2307.04725, 2023
Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. AnimateDiff: Animate your personalized text-to-image diffusion models without specific tuning.arXiv preprint arXiv:2307.04725, 2023
2023 arXiv
-
[27]
LTX-Video: Realtime video latent diffusion.arXiv preprint arXiv:2501.00103, 2024
Yoav HaCohen, Nisan Chiprut, Benny Brazowski, Daniel Shalem, Dudu Moshe, Eitan Richardson, Eran Levin, Guy Shiran, Nir Zabari, Ori Gordon, et al. LTX-Video: Realtime video latent diffusion.arXiv preprint arXiv:2501.00103, 2024
2024 arXiv
-
[28]
Harley, Zhaoyuan Fang, and Katerina Fragkiadaki
Adam W. Harley, Zhaoyuan Fang, and Katerina Fragkiadaki. Particle video revisited: Tracking through occlusions using point trajectories. InECCV, pages 59–75, 2022
2022
-
[29]
Latent video diffusion models for high-fidelity long video generation.arXiv preprint arXiv:2211.13221, 2022
Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity long video generation.arXiv preprint arXiv:2211.13221, 2022
2022 arXiv
-
[30]
Unsupervised semantic correspondence using stable diffusion.NeurIPS, 36:8266–8279, 2023
Eric Hedlin, Gopal Sharma, Shweta Mahajan, Hossam Isack, Abhishek Kar, Andrea Tagliasacchi, and Kwang Moo Yi. Unsupervised semantic correspondence using stable diffusion.NeurIPS, 36:8266–8279, 2023
2023
-
[31]
Denoising diffusion probabilistic models.NeurIPS, 33:6840– 6851, 2020
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models.NeurIPS, 33:6840– 6851, 2020
2020
-
[32]
Integrative feature and cost aggregation with transformers for dense correspondence.arXiv preprint arXiv:2209.08742, 2022
Sunghwan Hong, Seokju Cho, Seungryong Kim, and Stephen Lin. Integrative feature and cost aggregation with transformers for dense correspondence.arXiv preprint arXiv:2209.08742, 2022
2022 arXiv
-
[33]
Cost aggregation with 4D convolutional swin transformer for few-shot segmentation
Sunghwan Hong, Seokju Cho, Jisu Nam, Stephen Lin, and Seungryong Kim. Cost aggregation with 4D convolutional swin transformer for few-shot segmentation. InECCV, pages 108–126. Springer, 2022
2022
-
[34]
Unifying correspondence pose and nerf for generalized pose-free novel view synthesis
Sunghwan Hong, Jaewoo Jung, Heeseong Shin, Jiaolong Yang, Seungryong Kim, and Chong Luo. Unifying correspondence pose and nerf for generalized pose-free novel view synthesis. InCVPR, pages 20196–20206, 2024
2024
-
[35]
Deep matching prior: Test-time optimization for dense correspon- dence
Sunghwan Hong and Seungryong Kim. Deep matching prior: Test-time optimization for dense correspon- dence. InICCV, pages 9907–9917, 2021
2021
-
[36]
Neural matching fields: Implicit representation of matching fields for visual correspondence.NeurIPS, 35:13512–13526, 2022
Sunghwan Hong, Jisu Nam, Seokju Cho, Susung Hong, Sangryul Jeon, Dongbo Min, and Seungryong Kim. Neural matching fields: Implicit representation of matching fields for visual correspondence.NeurIPS, 35:13512–13526, 2022
2022
-
[37]
VBench: Comprehensive benchmark suite for video generative models
Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. VBench: Comprehensive benchmark suite for video generative models. InCVPR, pages 21807–21818, 2024
2024
-
[38]
Space-time correspondence as a contrastive random walk
Allan Jabri, Andrew Owens, and Alexei Efros. Space-time correspondence as a contrastive random walk. NeurIPS, 33:19545–19560, 2020
2020
-
[39]
Track4Gen: Teach- ing video diffusion models to track points improves video generation.arXiv preprint arXiv:2412.06016, 2024
Hyeonho Jeong, Chun-Hao Paul Huang, Jong Chul Ye, Niloy Mitra, and Duygu Ceylan. Track4Gen: Teach- ing video diffusion models to track points improves video generation.arXiv preprint arXiv:2412.06016, 2024. A.23
2024 arXiv
-
[40]
Appearance matching adapter for exemplar-based semantic image synthesis.arXiv preprint arXiv:2412.03150, 2024
Siyoon Jin, Jisu Nam, Jiyoung Kim, Dahyun Chung, Yeong-Seok Kim, Joonhyung Park, Heonjeong Chu, and Seungryong Kim. Appearance matching adapter for exemplar-based semantic image synthesis.arXiv preprint arXiv:2412.03150, 2024
2024 arXiv
-
[41]
CoTracker3: Simpler and better point tracking by pseudo-labelling real videos.arXiv preprint arXiv:2410.11831, 2024
Nikita Karaev, Iurii Makarov, Jianyuan Wang, Natalia Neverova, Andrea Vedaldi, and Christian Rup- precht. CoTracker3: Simpler and better point tracking by pseudo-labelling real videos.arXiv preprint arXiv:2410.11831, 2024
2024 arXiv
-
[42]
CoTracker: It is better to track together
Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. CoTracker: It is better to track together. InECCV, pages 18–35, 2024
2024
-
[43]
MUSIQ: Multi-scale image quality transformer
Junjie Ke, Qifei Wang, Yilin Wang, Peyman Milanfar, and Feng Yang. MUSIQ: Multi-scale image quality transformer. InICCV, pages 5148–5157, 2021
2021
-
[44]
Exploring temporally-aware features for point tracking.arXiv preprint arXiv:2501.12218, 2025
Inès Hyeonsu Kim, Seokju Cho, Jiahui Huang, Jung Yi, Joon-Young Lee, and Seungryong Kim. Exploring temporally-aware features for point tracking.arXiv preprint arXiv:2501.12218, 2025
2025 arXiv
-
[45]
MoDiTalker: Motion-disentangled diffusion model for high-fidelity talking head generation
Seyeon Kim, Siyoon Jin, Jihye Park, Kihong Kim, Jiyoung Kim, Jisu Nam, and Seungryong Kim. MoDiTalker: Motion-disentangled diffusion model for high-fidelity talking head generation. InAAAI, pages 4302–4310, 2025
2025
-
[46]
Berg, Wan-Yen Lo, et al
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, et al. Segment anything. InICCV, pages 4015–4026, 2023
2023
-
[47]
HunyuanVideo: A systematic framework for large video generative models.arXiv preprint arXiv:2412.03603, 2024
Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. HunyuanVideo: A systematic framework for large video generative models.arXiv preprint arXiv:2412.03603, 2024
2024 arXiv
-
[48]
Kling: Video generation by kuaishou, 2024
Kuaishou. Kling: Video generation by kuaishou, 2024. Accessed: 2025-03
2024
-
[49]
Efficient spatially sparse inference for conditional gans and diffusion models.NeurIPS, 35:28858–28873, 2022
Muyang Li, Ji Lin, Chenlin Meng, Stefano Ermon, Song Han, and Jun-Yan Zhu. Efficient spatially sparse inference for conditional gans and diffusion models.NeurIPS, 35:28858–28873, 2022
2022
-
[50]
Spatial-then-temporal self-supervised learning for video correspondence
Rui Li and Dong Liu. Spatial-then-temporal self-supervised learning for video correspondence. InCVPR, pages 2279–2288, 2023
2023
-
[51]
ReconX: Reconstruct any scene from sparse views with video diffusion model.arXiv preprint arXiv:2408.16767, 2024
Fangfu Liu, Wenqiang Sun, Hanyang Wang, Yikai Wang, Haowen Sun, Junliang Ye, Jun Zhang, and Yueqi Duan. ReconX: Reconstruct any scene from sparse views with video diffusion model.arXiv preprint arXiv:2408.16767, 2024
2024 arXiv
-
[52]
Sora: A review on background, technology, limitations, and opportunities of large vision models.arXiv preprint arXiv:2402.17177, 2024
Yixin Liu, Kai Zhang, Yuan Li, Zhiling Yan, Chujie Gao, Ruoxi Chen, Zhengqing Yuan, Yue Huang, Hanchi Sun, Jianfeng Gao, et al. Sora: A review on background, technology, limitations, and opportunities of large vision models.arXiv preprint arXiv:2402.17177, 2024
2024 arXiv
-
[53]
Not all diffusion model activations have been evaluated as discriminative features.NeurIPS, 37:55141–55177, 2025
Benyuan Meng, Qianqian Xu, Zitai Wang, Xiaochun Cao, and Qingming Huang. Not all diffusion model activations have been evaluated as discriminative features.NeurIPS, 37:55141–55177, 2025
2025
-
[54]
DreamMatcher: appearance matching self-attention for semantically-consistent text-to-image personalization
Jisu Nam, Heesu Kim, DongJae Lee, Siyoon Jin, Seungryong Kim, and Seunggyu Chang. DreamMatcher: appearance matching self-attention for semantically-consistent text-to-image personalization. InCVPR, pages 8100–8110, 2024
2024
-
[55]
Diffusion model for dense matching.arXiv preprint arXiv:2305.19094, 2023
Jisu Nam, Gyuseong Lee, Sunwoo Kim, Hyeonsu Kim, Hyoungwon Cho, Seyeon Kim, and Seungryong Kim. Diffusion model for dense matching.arXiv preprint arXiv:2305.19094, 2023
2023 arXiv
-
[56]
Visual Persona: Foundation model for full-body human customization.arXiv preprint arXiv:2503.15406, 2025
Jisu Nam, Soowon Son, Zhan Xu, Jing Shi, Difan Liu, Feng Liu, Aashish Misraa, Seungryong Kim, and Yang Zhou. Visual Persona: Foundation model for full-body human customization.arXiv preprint arXiv:2503.15406, 2025
2025 arXiv
-
[57]
DINOv2: Learning robust visual features without supervision.arXiv preprint arXiv:2304.07193, 2023
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. DINOv2: Learning robust visual features without supervision.arXiv preprint arXiv:2304.07193, 2023
2023 arXiv
-
[58]
Scalable diffusion models with transformers
William Peebles and Saining Xie. Scalable diffusion models with transformers. InICCV, pages 4195–4205, 2023
2023
-
[59]
SDXL: Improving latent diffusion models for high-resolution image synthesis.arXiv preprint arXiv:2307.01952, 2023
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis.arXiv preprint arXiv:2307.01952, 2023. A.24
2023 arXiv
-
[60]
Movie Gen: A cast of media foundation models, 2025
Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih-Yao Ma, Ching-Yao Chuang, et al. Movie Gen: A cast of media foundation models, 2025
2025
-
[61]
The 2017 DA VIS challenge on video object segmentation.arXiv preprint arXiv:1704.00675, 2017
Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine-Hornung, and Luc Van Gool. The 2017 DA VIS challenge on video object segmentation.arXiv preprint arXiv:1704.00675, 2017
2017 arXiv
-
[62]
Semantics meets temporal correspondence: Self- supervised object-centric learning in videos
Rui Qian, Shuangrui Ding, Xian Liu, and Dahua Lin. Semantics meets temporal correspondence: Self- supervised object-centric learning in videos. InICCV, pages 16675–16687, 2023
2023
-
[63]
Learning transferable visual models from natural language supervision
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InICML, pages 8748–8763, 2021
2021
-
[64]
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 21(140):1–67, 2020
2020
-
[65]
High-resolution image synthesis with latent diffusion models
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. InCVPR, pages 10684–10695, 2022
2022
-
[66]
Introducing Gen-3 Alpha, 2024
Runway. Introducing Gen-3 Alpha, 2024. Accessed: 2025-03
2024
-
[67]
Make-A-Video: Text-to-video generation without text-video data.arXiv preprint arXiv:2209.14792, 2022
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-A-Video: Text-to-video generation without text-video data.arXiv preprint arXiv:2209.14792, 2022
2022 arXiv
-
[68]
Denoising diffusion implicit models.arXiv preprint arXiv:2010.02502, 2020
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models.arXiv preprint arXiv:2010.02502, 2020
2010 arXiv
-
[69]
Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 568:127063, 2024
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 568:127063, 2024
2024
-
[70]
Dimen- sionX: Create any 3D and 4D scenes from a single image with controllable video diffusion.arXiv preprint arXiv:2411.04928, 2024
Wenqiang Sun, Shuo Chen, Fangfu Liu, Zilong Chen, Yueqi Duan, Jun Zhang, and Yikai Wang. Dimen- sionX: Create any 3D and 4D scenes from a single image with controllable video diffusion.arXiv preprint arXiv:2411.04928, 2024
2024 arXiv
-
[71]
Emergent correspondence from image diffusion.NeurIPS, 36:1363–1389, 2023
Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. Emergent correspondence from image diffusion.NeurIPS, 36:1363–1389, 2023
2023
-
[72]
RAFT: Recurrent all-pairs field transforms for optical flow
Zachary Teed and Jia Deng. RAFT: Recurrent all-pairs field transforms for optical flow. InECCV, pages 402–419, 2020
2020
-
[73]
GLU-Net: Global-local universal network for dense flow and correspondences
Prune Truong, Martin Danelljan, and Radu Timofte. GLU-Net: Global-local universal network for dense flow and correspondences. InCVPR, pages 6258–6268, 2020
2020
-
[74]
Learning accurate dense correspon- dences and when to trust them
Prune Truong, Martin Danelljan, Luc Van Gool, and Radu Timofte. Learning accurate dense correspon- dences and when to trust them. InCVPR, pages 5714–5724, 2021
2021
-
[75]
Gomez, Łukasz Kaiser, and Illia Polosukhin
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.NeurIPS, 30, 2017
2017
-
[76]
Barron, and Aleksander Holynski
Rundi Wu, Ruiqi Gao, Ben Poole, Alex Trevithick, Changxi Zheng, Jonathan T. Barron, and Aleksander Holynski. CAT4D: Create anything in 4D with multi-view video diffusion models.arXiv preprint arXiv:2411.18613, 2024
2024 arXiv
-
[77]
Video diffusion models are training-free motion interpreter and controller.arXiv preprint arXiv:2405.14864, 2024
Zeqi Xiao, Yifan Zhou, Shuai Yang, and Xingang Pan. Video diffusion models are training-free motion interpreter and controller.arXiv preprint arXiv:2405.14864, 2024
2024 arXiv
-
[78]
DynamiCrafter: Animating open-domain images with video diffusion priors
Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Wangbo Yu, Hanyuan Liu, Gongye Liu, Xintao Wang, Ying Shan, and Tien-Tsin Wong. DynamiCrafter: Animating open-domain images with video diffusion priors. InECCV, pages 399–417, 2024
2024
-
[79]
Rethinking self-supervised correspondence learning: A video frame-level similarity perspective
Jiarui Xu and Xiaolong Wang. Rethinking self-supervised correspondence learning: A video frame-level similarity perspective. InICCV, pages 10075–10085, 2021
2021
-
[80]
CogVideoX: Text-to-video diffusion models with an expert transformer.arXiv preprint arXiv:2408.06072, 2024
Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. CogVideoX: Text-to-video diffusion models with an expert transformer.arXiv preprint arXiv:2408.06072, 2024. A.25
2024 arXiv
-
[81]
A tale of two features: Stable diffusion complements dino for zero-shot semantic correspondence.NeurIPS, 36:45533–45547, 2023
Junyi Zhang, Charles Herrmann, Junhwa Hur, Luisa Polania Cabrera, Varun Jampani, Deqing Sun, and Ming-Hsuan Yang. A tale of two features: Stable diffusion complements dino for zero-shot semantic correspondence.NeurIPS, 36:45533–45547, 2023
2023
-
[82]
World-consistent video diffusion with explicit 3D modeling.arXiv preprint arXiv:2412.01821, 2024
Qihang Zhang, Shuangfei Zhai, Miguel Angel Bautista, Kevin Miao, Alexander Toshev, Joshua Susskind, and Jiatao Gu. World-consistent video diffusion with explicit 3D modeling.arXiv preprint arXiv:2412.01821, 2024
2024 arXiv
-
[83]
Open-Sora: Democratizing efficient video production for all.arXiv preprint arXiv:2412.20404, 2024
Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-Sora: Democratizing efficient video production for all.arXiv preprint arXiv:2412.20404, 2024. A.26
2024 arXiv
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.