Pith. sign in

REVIEW 4 major objections 5 minor 69 references

From Image to Video: An Empirical Study of Diffusion Representations

T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Training a diffusion model to generate video instead of images produces better features on all ten vision tasks tested.

desk verdict The matched-architecture comparison is a real advance, but the missing I-WALT training-budget details leave the causal story one checkable fact short. read the letter →

arxiv 2502.07001 v2 pith:2UL25M2Q submitted 2025-02-10 cs.CV cs.AIcs.LG

Pedro Vélez , Luisa F. Polanía , Yi Yang , Chuhan Zhang , Rishabh Kabra , Anurag Arnab , Mehdi S. M. Sajjadi This is my paper · ORCID
classification cs.CVcs.AIcs.LG
keywords diffusionmodelsvideogenerationimagerepresentationlearningtemporalattentionactionrecognitiondepthestimationpointtracking
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether a diffusion model learns better visual representations when its generation target is video rather than still images. Using the WALT architecture, which can run in both image and video modes with matched parameters and training data, the authors compare frozen features from a video-trained model (V-WALT) with an image-trained counterpart (I-WALT) on ten readout tasks. The video-trained model wins on every task, with the largest gains on motion- and geometry-heavy benchmarks—point tracking (+68%), camera pose (+60%), Something-Something-v2 action recognition (+42%), and box tracking (+23%)—and near-parity on purely semantic classification such as ImageNet (+1.8%) and Places365 (+0.6%). The authors conclude that the temporal denoising objective itself, rather than model scale or data, drives better spatiotemporal features, and they characterize where these features live across noise levels, network depth, model size, and training progress.

What carries the argument

The central mechanism is the controlled image-video comparison made possible by the WALT architecture, a latent diffusion transformer that can switch between video mode (V-WALT), where alternating spatio-temporal-window and spatial-window attention blocks model motion, and image mode (I-WALT), where the spatio-temporal blocks are replaced by windowed spatial blocks of equal parameter count, so that only temporal attention and the generation objective differ. The evaluating machinery is a probing framework: input latents are given a small noise at timestep $t$, the frozen denoiser is run once with a null text embedding, activations from block $l$ are read out by task-specific cross-attention heads, and performance is measured as relative change $(x_V - x_I)/x_I$ against the image baseline. The framework includes tuning of $t$ and $l$, with small noise ($t \approx 200$) and mid-to-late blocks (about two-thirds depth) giving the best features for most tasks, while tracking prefers lower noise and earlier blocks.

What would settle it

Train I-WALT and V-WALT from identical data and compute budgets, match their generation quality (e.g., equal FVD/FID), and probe each model at its own optimal noise level and block index; if the video model's advantage on motion tasks disappears, the claim that the temporal objective drives the features is false. A second check: train a video model with temporal attention disabled but the same data and parameters; if it matches V-WALT, the temporal objective is not the causal driver. Releasing I-WALT would also let an independent group verify the ten readout numbers and the +68% point-tracking gap.

Watch

Extended reading notes

Core claim

The central claim is that video diffusion representation learning consistently surpasses image diffusion representation learning under a controlled comparison: the same architecture, parameter count, training settings, and data distribution, differing only in whether the model denoises video latents with spatio-temporal attention or denoises image latents with matched windowed spatial attention. The evidence comes from a probing pipeline that adds a small amount of noise to input latents, runs a single forward pass with a null text embedding, extracts activations from transformer blocks, and trains lightweight readout heads for image classification (ImageNet, Places365, iNaturalist-2018), action recognition (Kinetics-400/700, Something-Something-v2), monocular depth, relative camera pose, and point and box tracking. V-WALT outperforms I-WALT on all ten tasks, with relative gains from +0.6% (Places365) to +68% (PointTracks), and the gain ordering tracks how much a task depends on motion and 3D structure. The paper reads this as evidence that the temporal generation objective teaches the model to represent motion and spatial layout, and it notes that image diffusion's known strength at point correspondence does not transfer to point tracking, which demands precise localization.

Load-bearing premise

The central claim depends on I-WALT being a fair, equally well-trained image twin of V-WALT; because I-WALT was trained by the authors and is not publicly released, a skeptic cannot rule out that differences in training quality or effective data distribution, rather than the video objective, produce the observed gaps.

Editorial extensions

If this is right

  • Video generation pre-training can serve as a spatiotemporal representation learner: the same frozen backbone reads out for recognition, depth, pose, and tracking without task-specific pre-training.
  • The gains are concentrated where tasks need motion and 3D geometry (point tracking +68%, camera pose +60%, SSv2 +42%), while semantic classification changes little (ImageNet +1.8%, Places365 +0.6%), so the temporal objective is not a universal feature booster.
  • Small noise during feature extraction helps most readouts ($t=200$ optimal for most tasks, $t=0$ for box tracking, $t=100$ for point tracking), and the best blocks sit at roughly two-thirds depth, giving concrete guidance for probing diffusion backbones.
  • Scaling the video model from 284M to 1.9B parameters improves most tasks, especially large-class classification, but V-WALT still trails DINOv2 and SigLIP on semantic tasks, marking a boundary of diffusion-based features.
  • Tracking and depth features mature early in training and camera pose performance declines after 26% of the schedule, so earlier checkpoints can be better than the final model for some downstream uses.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the temporal objective is the causal driver, mixing image and video batches during training should yield a smooth interpolation of downstream gains; the paper does not test this, but it is a direct way to separate the objective from the data distribution.
  • Editorial inference: the near-parity on ImageNet and Places365 suggests a combined image+video objective could retain semantic accuracy while gaining motion and geometry skills, a training recipe the paper leaves implicit.
  • Editorial inference: the headline comparison does not explicitly state that both models were probed at the same noise level and block index; a reproducibility check that probes each model at its own optimal $(t, l)$ would confirm the ranking is not an artifact of a fixed probing choice.
  • Editorial inference: the finding that image diffusion features excel at semantic correspondence but not at precise point tracking could be tested by evaluating I-WALT and V-WALT on a semantic-correspondence benchmark (e.g., DIFT-style sparse matching) where the image model should close the gap.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper compares the visual representations of a diffusion transformer trained for video generation (V-WALT) with those of the same architecture trained for image generation (I-WALT), both operating in the latent space of the MAGVIT-v2 tokenizer. Using a frozen-backbone probing framework with task-specific readout heads, the authors report that V-WALT outperforms I-WALT on ten downstream tasks spanning image classification, action recognition, depth estimation, relative camera pose estimation, and point/box tracking. They also analyze the effect of the noise timestep and transformer block used for feature extraction, study the relationship between pretraining budget and downstream performance, and compare V-WALT against established self-supervised models such as DINOv2, SigLIP, MAE, and JEPA. The central claim is that the temporal generation objective, rather than architecture scale or data, drives the representation gains.

Significance. If the central comparison is sound, this is the first direct controlled comparison of image versus video diffusion objectives for visual understanding, and it provides useful evidence that a temporal denoising objective yields stronger spatiotemporal features for perception tasks. The experiment is thoughtfully designed in several respects: V-WALT and I-WALT share the same architecture and parameter count, use the same internal data source, use frozen backbones, and are evaluated with an identical readout protocol per task. The paper also includes useful ablations of noise level and block depth, a training-budget study, and a comparison with common visual representation models under the same readout setup. The main significance hinges on whether I-WALT is a faithful, equally trained image counterpart, because the headline numbers (e.g., PointTracks +68%, Cam. Pose +60%, SSv2 +42%) are attributed to the video objective. At present, the evidence for that attribution is incomplete, and the lack of error bars makes the smaller deltas difficult to interpret.

major comments (4)
  1. [Sec. 3.3] The statement 'The training settings of I-WALT are the same as WALT' is the only description of I-WALT pretraining; the paper does not report I-WALT's total optimizer steps, epochs, batch size, dataset composition, or a convergence metric such as final training loss or FID. Appendix E documents only readout training, and Appendix A's Fig. 9 is qualitative. Because the central claim is that the video generation objective, not training investment, causes the observed representation gains, the comparison must be matched on pretraining budget. Without this evidence, the large deltas (PointTracks +68%, Cam. Pose +60%, SSv2 +42%) could reflect an undertrained I-WALT rather than the temporal objective.
  2. [Sec. 4.2, Fig. 1, Appendix B Tables 3-4] All reported numbers are single runs without error bars or repeated seeds. The smallest gaps (Places365 +0.6%, ImageNet +1.8%) are within the range typically expected from readout-initialization noise, so the claim that V-WALT 'consistently outperforms' I-WALT across all ten tasks is not statistically supported for those tasks. The authors should report at least the variance over readout training seeds, and ideally over a small number of backbone training runs.
  3. [Sec. 4.3 and Appendix A] The paper never states which noise timestep t and block index l were used to produce the headline V-WALT versus I-WALT comparison in Fig. 1 and Tables 3-4. Section 4.3 reports ablations for V-WALT and Appendix A reports ablations for I-WALT, but the protocol for selecting the operating point (per-task optimum, global optimum, or a fixed t=200, l=16) is not specified. If different (t,l) values were selected per task or per model, the comparison could be confounded; the exact protocol must be stated.
  4. [Sec. 3.3] The data control is incomplete: I-WALT is trained on randomly extracted single frames from the same internal dataset, while V-WALT is trained on 17-frame clips. The paper does not report how many frames per video were sampled for I-WALT or how the image/video mixture compares between the two training runs. If I-WALT sees fewer unique frames or a different image/video ratio per optimizer step, the comparison conflates the objective with the data schedule. The authors should specify the exact sampling and training schedule for I-WALT.
minor comments (5)
  1. [Fig. 7 and Table 4] The figure legend labels a model as 'V-WALT-3B' while the text and Table 4 refer to the same model as 'V-WALT 1.9B'; these should be reconciled.
  2. [References] Reference [2] is listed as 'Anonymous authors' and 'Under review'; if the related work has since been published or de-anonymized, the citation should be updated.
  3. [Throughout] The notation 'V-W ALT', 'I-W ALT', and 'W ALT' contains spurious spaces from line breaks; the authors should use a consistent macro or formatting to avoid ambiguity.
  4. [Sec. 3.3 and Appendix E] The claim that I-WALT and V-WALT 'share the same architecture' should be accompanied by a precise statement of how the spatio-temporal blocks are replaced by spatial blocks while keeping parameter counts, position-encoding, and attention-mask implementations identical.
  5. [Fig. 4 caption] The phrase 'In the 4 brick videos' is unclear; it should be reworded to describe the four manipulated videos or the four panels shown.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the central video-vs-image comparison is a controlled empirical measurement against external benchmarks, not a derivation from fitted parameters or self-citations.

full rationale

The paper's central claim is an empirical comparison of frozen V-WALT and I-WALT representations measured through task-specific readouts (Sec. 3.4) on external benchmarks such as ImageNet, Places365, Kinetics, SSv2, ScanNet, RealEstate10K, and Perception Test. The only formula defining relative performance, (xV - xI)/xI, is a reporting normalization rather than a derivation, and no fitted parameter is renamed as a prediction. The in-house I-WALT baseline is trained for controlled comparison, but the conclusion is not defined in terms of I-WALT's training curve or any fitted value; the missing pretraining recipe and convergence details are an evidence/completeness limitation and a potential confound, not circularity. Self-citations to WALT [22], the 4DS probing setup [11], MooG [55], and SRT [45] supply the architecture, readout heads, and evaluation methodology, but none of these cited works asserts the paper's outcome, so the self-citations are not load-bearing. There is no imported uniqueness theorem, no ansatz smuggled in via citation, and no known result merely renamed. The comparison is therefore self-contained as a measurement, and the score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim is an empirical comparison, so the ledger records evaluation choices (noise level, block, training settings) that the result depends on. There are no fitted constants in a derivation sense; the main free choices are the probing operating points, which are not fully specified for the headline numbers.

free parameters (2)
  • noise timestep t = 0 to 200, task dependent
    Selected from a small grid in Sec 4.3; the headline comparison numbers do not state the per-model settings.
  • block index l = 11 to 16, task dependent
    Selected from the 24 transformer blocks; not stated whether the same block was used for both models in the headline results.
assumptions (4)
  • domain assumption Frozen single-pass denoiser features at one noise level are a valid representation for downstream tasks
    Standard probing assumption from the V-JEPA and 4DS literature; adopted here without re-validation for diffusion models.
  • domain assumption I-WALT and V-WALT differ only in the generation objective, with matched architecture, parameters, dataset, and training settings
    Stated in Sec 3.3; I-WALT was trained for this paper and is not publicly available.
  • domain assumption The internal image and video dataset is representative and the frame sampling does not introduce a confound
    Used for both models; the dataset is not released, so its composition cannot be checked.
  • domain assumption Readout heads are trained to comparable convergence for both backbones
    Training schedules are fixed per task (Table 7), but convergence and run-to-run variance are not reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of From Image to Video: An Empirical Study of Diffusion Representations." pith.science (2026). https://pith.science/paper/2UL25M2Q

@misc{pith2026250207001,
  author       = {Pith},
  title        = {Pith review of: From Image to Video: An Empirical Study of Diffusion Representations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2UL25M2Q}},
  note         = {Machine review of arXiv:2502.07001}
}
read the original abstract

Diffusion models have revolutionized generative modeling, enabling unprecedented realism in image and video synthesis. This success has sparked interest in leveraging their representations for visual understanding tasks. While recent works have explored this potential for image generation, the visual understanding capabilities of video diffusion models remain largely uncharted. To address this gap, we systematically compare the same model architecture trained for video versus image generation, analyzing the performance of their latent representations on various downstream tasks including image classification, action recognition, depth estimation, and tracking. Results show that video diffusion models consistently outperform their image counterparts, though we find a striking range in the extent of this superiority. We further analyze features extracted from different layers and with varying noise levels, as well as the effect of model size and training budget on representation and generation quality. This work marks the first direct comparison of video and image diffusion objectives for visual understanding, offering insights into the role of temporal information in representation learning.

Figures

Figures reproduced from arXiv: 2502.07001 by the authors.

Figure 1
Figure 1. Video vs. image diffusion representations – The diffu￾sion model V-WALT trained on generating videos learns better fea￾tures than the same model trained on generating images (I-WALT, normalized to 100% here) as measured across a range of readout tasks. See Sec. 4.2 for details. Meanwhile, generative models have demonstrated an un￾precedented ability to synthesize novel, photo-realistic im￾agery [27, 29]. Among gener… view at source ↗
Figure 2
Figure 2. Probing architecture – We feed videos through the model and extract (frozen) intermediate features. Cross-attention modules then read out the label for the downstream tasks. single diffusion model is not ideal, the same fair compari￾son is not feasible with open-sourced video diffusion mod￾els given the architecture size disparities with their image counterparts. For example, there is a significant architec￾ture siz… view at source ↗
Figure 3
Figure 3. Feature visualization – We show the major PCA com￾ponent for the two models across a range of DAVIS videos. While I-WALT is sensitive to semantically important areas of the scene (e.g., all people in the second column), V-WALT is much more sensitive to the areas that experience motion within the video (e.g., only the wrestlers in the same video). highlighting how I-WALT and V-WALT differ in represent￾ing motion. Bot… view at source ↗
Figures from the paper (7 more)
Figure 5
Figure 5. Figure 5: Influence of Noise and Block Choice on Readout Performance – Relative change in downstream task performance when probing different noise levels (left, fixed block l = 16) and intermediate WALT blocks (right, fixed noise t = 200). Values below -10% are excluded for clar…
Figure 6
Figure 6. Figure 6: Impact of pre-training progress on downstream task performance – Recognition tasks generally improve with longer training, while tasks like tracking and depth estimation show opti￾mal performance at earlier stages. Performance is evaluated across a range of tasks and c…
Figure 8
Figure 8. Figure 8: Influence of Noise and Block Choice on Readout Performance of I-WALT – Relative change in downstream task performance when probing different noise levels (left, fixed block l = 16) and intermediate WALT blocks (right, fixed noise t = 200). Values below -10% are exclude…
Figure 9
Figure 9. Figure 9: Impact of pre-training progress on downstream task performance of I-WALT – Recognition tasks generally improve with longer training, while tasks like tracking and depth estimation show optimal performance at earlier stages. Performance is eval￾uated across a range of t…
Figure 10
Figure 10. Figure 10: Depth predictions from I-WALT and V-WALT – RGB, Ground Truth, and Predictions of I-WALT and V-WALT (284M and 1.9B) models. The results of V-WALT 1.9B are resized to a square aspect ratio for visualization purposes. 6 [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: Object Tracks predictions from I-WALT and V-WALT – Ground Truth, and Predictions of I-WALT and V-WALT (284M and 1.9B) models. The results of V-WALT 1.9B are resized to a square aspect ratio for visualization purposes. 7 [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]
Figure 12
Figure 12. Figure 12: Action recognition predictions from SSv2 using I-WALT and V-WALT – The top row displays a single frame from a dataset sample. Frames were manually selected to best showcase the corresponding label. 8 [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

69 extracted references · 45 canonical work pages

  1. [1]

    Self-supervised learning from images with a joint-embedding predictive architecture

    Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bo- janowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embedding predictive architecture. In CVPR, 2023. 1, 8

    work page 2023

  2. [2]

    Video diffusion models learn the struc- ture of the dynamic world

    Anonymous authors. Video diffusion models learn the struc- ture of the dynamic world. In Submitted to ICLR, 2024. Un- der review. 2

    work page 2024

  3. [3]

    Learning by recon- struction produces uninformative features for perception

    Randall Balestriero and Yann LeCun. Learning by recon- struction produces uninformative features for perception. arXiv preprint arXiv:2402.11337, 2024. 7

    arXiv 2024

  4. [4]

    Label-efficient se- mantic segmentation with diffusion models

    Dmitry Baranchuk, Andrey V oynov, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Label-efficient se- mantic segmentation with diffusion models. In ICLR, 2022. 2

    work page 2022

  5. [5]

    Revisiting feature prediction for learn- ing visual representations from video

    Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mahmoud Assran, and Nicolas Ballas. Revisiting feature prediction for learn- ing visual representations from video. arXiv preprint arXiv:2404.08471, 2024. 2, 4, 6, 8, 3

    arXiv 2024

  6. [6]

    Stable Video Diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable Video Diffusion: Scaling latent video diffusion models to large datasets. In arXiv preprint arXiv:2311.15127, 2023. 2, 3

    arXiv 2023

  7. [7]

    Deep regression on manifolds: a 3D rota- tion case study

    Romain Br ´egier. Deep regression on manifolds: a 3D rota- tion case study. In 3DV, 2021. 5, 4

    work page 2021

  8. [8]

    Emerg- ing properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv ´e J´egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerg- ing properties in self-supervised vision transformers. In ICCV, pages 9650–9660, 2021. 1, 7

    work page 2021

Show all 69 references
  1. [9]

    Quo vadis, action recognition? a new model and the kinetics dataset

    Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In CVPR,

  2. [10]

    A short note on the kinetics-700 human action dataset

    Joao Carreira, Eric Noland, Chloe Hillier, and Andrew Zis- serman. A short note on the kinetics-700 human action dataset. arXiv preprint arXiv:1907.06987, 2019. 4

    1907 arXiv

  3. [11]

    Jo ˜ao Carreira, Dilara Gokay, Michael King, Chuhan Zhang, Ignacio Rocco, Aravindh Mahendran, Thomas Albert Keck, Joseph Heyward, Skanda Koppula, Etienne Pot, Goker Erdo- gan, Yana Hasson, Yi Yang, Klaus Greff, Guillaume Le Mo- ing, Sjoerd van Steenkiste, Daniel Zoran, Drew A....

    2024 arXiv

  4. [12]

    Ting Chen, Lala Li, Saurabh Saxena, Geoffrey Hinton, and David J. Fleed. A generalist framework for panoptic seg- mentation of images and videos. In ICCV, 2023. 2

    2023

  5. [13]

    Pali-3 vision language models: Smaller, faster, stronger

    Xi Chen, Xiao Wang, Lucas Beyer, Alexander Kolesnikov, Jialin Wu, Paul V oigtlaender, Basil Mustafa, Sebastian Goodman, Ibrahim Alabdulmohsin, Piotr Padlewski, et al. Pali-3 vision language models: Smaller, faster, stronger. arXiv preprint arXiv:2310.09199, 2023. 7

    arXiv

  6. [14]

    Text-to-image diffusion mod- els are zero shot classifiers

    Kevin Clark and Priyank Jaini. Text-to-image diffusion mod- els are zero shot classifiers. In NeurIPS, 2024. 2

    2024

  7. [15]

    Chang, Manolis Savva, Maciej Hal- ber, Thomas Funkhouser, and Matthias Nießner

    Angela Dai, Angel X. Chang, Manolis Savva, Maciej Hal- ber, Thomas Funkhouser, and Matthias Nießner. ScanNet: Richly-annotated 3D reconstructions of indoor scenes. In CVPR, 2017. 5, 4

    2017

  8. [16]

    Depth map prediction from a single image using a multi-scale deep net- work

    David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from a single image using a multi-scale deep net- work. NeurIPS, 2014. 5

    2014

  9. [17]

    Taming transformers for high-resolution image synthesis

    Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR,

  10. [18]

    Masked autoencoders as spatiotemporal learners

    Christoph Feichtenhofer, Yanghao Li, Kaiming He, et al. Masked autoencoders as spatiotemporal learners. NeurIPS,

  11. [19]

    Diffusion mod- els and representation learning: A survey

    Michael Fuest, Pingchuan Ma, Ming Gui, Johannes S Fis- cher, Vincent Tao Hu, and Bjorn Ommer. Diffusion mod- els and representation learning: A survey. arXiv preprint arXiv:2407.00783, 2024. 1, 2

    2024 arXiv

  12. [20]

    Something Something

    Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michal- ski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, Florian Hoppe, Christian Thurau, Ingo Bax, and Roland Memisevic. The “Something Something” Video Dat...

    2017

  13. [21]

    Kubric: A scalable dataset generator

    Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David J Fleet, Dan Gnanapra- gasam, Florian Golemo, Charles Herrmann, et al. Kubric: A scalable dataset generator. In CVPR, 2022. 5

    2022

  14. [22]

    Photorealistic video generation with diffusion models

    Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Li Fei-Fei, Irfan Essa, Lu Jiang, and Jos ´e Lezama. Photorealistic video generation with diffusion models. In ECCV, 2024. 2, 3, 8

    2024

  15. [23]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll´ar, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022. 1, 8

    2022

  16. [24]

    Unsupervised keypoints from pretrained diffusion models

    Eric Hedlin, Gopal Sharma, Shweta Mahajan, Xingzhe He, Hossam Isack, Abhishek Kar, Helge Rhodin, Andrea Tagliasacchi, and Kwang Moo Yi. Unsupervised keypoints from pretrained diffusion models. In CVPR, 2024. 1, 6

    2024

  17. [25]

    Denoising diffu- sion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffu- sion probabilistic models. In NeurIPS, 2020. 1, 2

    2020

  18. [26]

    Depthcrafter: Generating consistent long depth sequences for open-world videos

    Wenbo Hu, Xiangjun Gao, Xiaoyu Li, Sijie Zhao, Xiaodong Cun, Yong Zhang, Long Quan, and Ying Shan. Depthcrafter: Generating consistent long depth sequences for open-world videos. arXiv preprint arXiv:2409.02095, 2024. 2

    2024 arXiv

  19. [27]

    Elucidating the design space of diffusion-based generative models

    Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. NeurIPS, 2022. 1, 2 9

    2022

  20. [28]

    Your diffusion model is secretly a zero-shot classifier

    Alexander C Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak. Your diffusion model is secretly a zero-shot classifier. In ICCV, 2023. 1

    2023

  21. [29]

    Sora: A review on background, technology, limitations, and opportunities of large vision models

    Yixin Liu, Kai Zhang, Yuan Li, Zhiling Yan, Chujie Gao, Ruoxi Chen, Zhengqing Yuan, Yue Huang, Hanchi Sun, Jian- feng Gao, et al. Sora: A review on background, technology, limitations, and opportunities of large vision models. arXiv preprint arXiv:2402.17177, 2024. 1, 2

    2024 arXiv

  22. [30]

    Diffusion hyperfeatures: Search- ing through time and space for semantic correspondence

    Grace Luo, Lisa Dunlap, Dong Huk Park, Aleksander Holyn- ski, and Trevor Darrell. Diffusion hyperfeatures: Search- ing through time and space for semantic correspondence. In NeurIPS, 2023. 2, 7

    2023

  23. [31]

    Understanding deep image representations by inverting them

    Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by inverting them. In CVPR,

  24. [32]

    Lexicon3d: Probing vi- sual foundation models for complex 3d scene understanding

    Yunze Man, Shuhong Zheng, Zhipeng Bao, Martial Hebert, Liangyan Gui, and Yu-Xiong Wang. Lexicon3d: Probing vi- sual foundation models for complex 3d scene understanding. In The Thirty-eighth Annual Conference on Neural Informa- tion Processing Systems, 2024. 2

    2024

  25. [33]

    NeRF: Representing scenes as neural radiance fields for view syn- thesis

    Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view syn- thesis. In ECCV, 2020. 5

    2020

  26. [34]

    Diffusion models beat GANs on image classification

    Soumik Mukhopadhyay, Matthew Gwilliam, Vatsal Agar- wal, Namitha Padmanabhan, Archana Swaminathan, Srinidhi Hegde, Tianyi Zhou, and Abhinav Shrivastava. Diffusion models beat GANs on image classification. arXiv preprint arXiv:2307.08702, 2023. 2, 7

    2023 arXiv

  27. [35]

    DiffTAD: Temporal action detection with proposal denoising diffusion

    Sauradip Nag, Xiatian Zhu, Jiankang Deng, Yi-Zhe Song, and Tao Xiang. DiffTAD: Temporal action detection with proposal denoising diffusion. In ICCV, 2023. 2

    2023

  28. [36]

    Maxime Oquab, Timoth ´ee Darcet, Th´eo Moutakanni, Huy V . V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael ...

    2024

  29. [37]

    Self-supervised video pretraining yields robust and more human-aligned visual representations

    Nikhil Parthasarathy, SM Eslami, Joao Carreira, and Olivier Henaff. Self-supervised video pretraining yields robust and more human-aligned visual representations. NeurIPS, 2023. 2

    2023

  30. [38]

    Per- ception Test: A diagnostic benchmark for multimodal video models

    Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Re- casens, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Mateusz Malinowski, Yi Yang, Carl Doersch, et al. Per- ception Test: A diagnostic benchmark for multimodal video models. NeurIPS, 2024. 5, 4

    2024

  31. [39]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, 2023. 2

    2023

  32. [40]

    The 2017 DA VIS challenge on video object segmentation

    Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Ar- bel´aez, Alexander Sorkine-Hornung, and Luc Van Gool. The 2017 DA VIS challenge on video object segmentation. arXiv:1704.00675, 2017. 5

    2017 arXiv

  33. [41]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In ICML, 2021. 1

    2021

  34. [42]

    High-resolution image syn- thesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨orn Ommer. High-resolution image syn- thesis with latent diffusion models. In CVPR, 2022. 2, 3

    2022

  35. [43]

    U- Net: Convolutional networks for biomedical image segmen- tation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U- Net: Convolutional networks for biomedical image segmen- tation. In MICCAI, 2015. 2

    2015

  36. [44]

    Berg, and Li Fei-Fei

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, San- jeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet large scale visual recognition chal- lenge. IJCV, 2015. 4, 2

    2015

  37. [45]

    Scene representation transformer: Geometry-free novel view syn- thesis through set-latent scene representations

    Mehdi SM Sajjadi, Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani V ora, Mario Luˇci´c, Daniel Duckworth, Alexey Dosovitskiy, et al. Scene representation transformer: Geometry-free novel view syn- thesis through set-latent scene representations. In CVPR,

  38. [46]

    Only time can tell: Discovering temporal data for temporal modeling

    Laura Sevilla-Lara, Shengxin Zha, Zhicheng Yan, Vedanuj Goswami, Matt Feiszli, and Lorenzo Torresani. Only time can tell: Discovering temporal data for temporal modeling. In WACV, 2021. 6

    2021

  39. [47]

    MonoDiffusion: Self-supervised monocular depth estimation using diffusion model

    Shuwei Shao, Zhongcai Pei, Weihai Chen, Dingchi Sun, Peter CY Chen, and Zhengguo Li. MonoDiffusion: Self-supervised monocular depth estimation using diffusion model. arXiv preprint arXiv:2311.07198, 2023. 1

    2023 arXiv

  40. [48]

    Deep unsupervised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML, 2015. 3

    2015

  41. [49]

    Scalability in perception for autonomous driving: Waymo Open Dataset

    Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aur ´elien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Aleksei Timofeev, Scott Et- tinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Yu Zh...

    2020

  42. [50]

    Emergent correspondence from image diffusion

    Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. Emergent correspondence from image diffusion. NeurIPS, 2023. 1, 2, 6, 7

    2023

  43. [51]

    VideoMAE: Masked autoencoders are data-efficient learners for self-supervised video pre-training

    Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. VideoMAE: Masked autoencoders are data-efficient learners for self-supervised video pre-training. NeurIPS, 2022. 2, 8

    2022

  44. [52]

    To- wards accurate generative models of video: A new metric & challenges

    Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. To- wards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01717, 2018. 7, 1

    2018 arXiv

  45. [53]

    Neural discrete representation learning

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. NeurIPS, 2017. 3

    2017

  46. [54]

    The iNaturalist species classification and detection dataset

    Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The iNaturalist species classification and detection dataset. In CVPR, 2018. 4, 2 10

    2018

  47. [55]

    Hudson, Thomas Albert Keck, Joao Carreira, Alexey Doso- vitskiy, Mehdi S

    Sjoerd van Steenkiste, Daniel Zoran, Yi Yang, Yulia Rubanova, Rishabh Kabra, Carl Doersch, Dilara Gokay, Joseph Heyward, Etienne Pot, Klaus Greff Greff, Drew A. Hudson, Thomas Albert Keck, Joao Carreira, Alexey Doso- vitskiy, Mehdi S. M. Sajjadi, and Thomas Kipf. Moving off- t...

  48. [56]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017. 3

    2017

  49. [57]

    VideoMAE v2: Scaling video masked autoencoders with dual masking

    Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yi- nan He, Yi Wang, Yali Wang, and Yu Qiao. VideoMAE v2: Scaling video masked autoencoders with dual masking. In CVPR, pages 14549–14560, 2023. 2

    2023

  50. [58]

    Controlling space and time with dif- fusion models

    Daniel Watson, Saurabh Saxena, Lala Li, Andrea Tagliasac- chi, and David J Fleet. Controlling space and time with dif- fusion models. arXiv preprint arXiv:2407.07860, 2024. 5

    2024 arXiv

  51. [59]

    Denoising diffusion autoencoders are unified self-supervised learners

    Weilai Xiang, Hongyu Yang, Di Huang, and Yunhong Wang. Denoising diffusion autoencoders are unified self-supervised learners. In ICCV, 2023. 2, 7

    2023

  52. [60]

    Open-vocabulary panop- tic segmentation with text-to-image diffusion models

    Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiao- long Wang, and Shalini De Mello. Open-vocabulary panop- tic segmentation with text-to-image diffusion models. In CVPR, 2023. 2

    2023

  53. [61]

    Diffusion Model as Rep- resentation Learner

    Xingyi Yang and Xinchao Wang. Diffusion Model as Rep- resentation Learner. In ICCV, 2023. 2

    2023

  54. [62]

    Gundavarapu, Luca Ver- sari, Kihyuk Sohn, David Minnen, Yong Cheng, Vigh- nesh Birodkar, Agrim Gupta, Xiuye Gu, Alexander G

    Lijun Yu, Jos ´e Lezama, Nitesh B. Gundavarapu, Luca Ver- sari, Kihyuk Sohn, David Minnen, Yong Cheng, Vigh- nesh Birodkar, Agrim Gupta, Xiuye Gu, Alexander G. Hauptmann, Boqing Gong, Ming-Hsuan Yang, Irfan Essa, David A. Ross, and Lu Jiang. Language model beats diffu- sion – ...

    2024

  55. [63]

    A tale of two features: Stable diffusion complements DINO for zero-shot semantic correspondence

    Junyi Zhang, Charles Herrmann, Junhwa Hur, Luisa Pola- nia Cabrera, Varun Jampani, Deqing Sun, and Ming-Hsuan Yang. A tale of two features: Stable diffusion complements DINO for zero-shot semantic correspondence. NeurIPS,

  56. [64]

    A survey of diffusion based image generation models: Issues and their solutions

    Tianyi Zhang, Zheng Wang, Jing Huang, Mohiud- din Muhammad Tasnim, and Wei Shi. A survey of diffusion based image generation models: Issues and their solutions. arXiv preprint arXiv:2308.13142, 2023. 2

    2023 arXiv

  57. [65]

    Long Zhao, Nitesh Bharadwaj Gundavarapu, Liangzhe Yuan, Hao Zhou, Shen Yan, Jennifer J. Sun, Luke Friedman, Rui Qian, Tobias Weyand, Yue Zhao, Rachel Hornung, Flo- rian Schroff, Ming-Hsuan Yang, David A Ross, Huisheng Wang, Hartwig Adam, Mikhail Sirotenko, Ting Liu, and Bo- qi...

    2024

  58. [66]

    Unleashing text-to-image diffusion models for visual perception

    Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, and Jiwen Lu. Unleashing text-to-image diffusion models for visual perception. In ICCV, 2023. 2

    2023

  59. [67]

    Places: A 10 million image database for scene recognition

    Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE TPAMI, 2017. 4, 2

    2017

  60. [68]

    Stereo magnification: Learning view syn- thesis using multiplane images

    Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view syn- thesis using multiplane images. ACM TOG, 2018. 5, 4

    2018

  61. [69]

    squeezing some- thing

    Zixin Zhu, Xuelu Feng, Dongdong Chen, Junsong Yuan, Chunming Qiao, and Gang Hua. Exploring pre-trained text- to-video diffusion models for referring video object segmen- tation. arXiv preprint arXiv:2403.12042, 2024. 2 11 From Image to Video: An Empirical Study of Diffusion Re...

    2024 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.