Pith. sign in

REVIEW 4 major objections 5 minor 38 references

Aligning video diffusion transformer features with a fused DINOv2 and SAM2.1 Hiera target improves generation quality and converges at least 2.5x faster than the vanilla baseline.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 18:52 UTC pith:SQQ23FVN

load-bearing objection The alignment idea transfers to video cleanly and the experiments are honest, but the headline efficiency claim rests on step counts, not wall-clock time, so it is not proven yet. the 4 major comments →

arxiv 2509.09547 v1 pith:SQQ23FVN submitted 2025-09-11 cs.CV cs.AI

Improving Video Diffusion Transformer Training by Multi-Feature Fusion and Alignment from Self-Supervised Vision Encoders

Dohun Lee , Hyeonho Jeong , Jiwook Kim , Duygu Ceylan , Jong Chul Ye This is my paper
classification cs.CV cs.AI
keywords video diffusion transformersfeature alignmentself-supervised vision encodersDINOv2SAM2IICRtraining accelerationfrequency complementarity
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that video diffusion transformers learn better and faster when their intermediate patch features are pulled toward representations from pretrained image encoders. It introduces a metric, IICR, that scores encoders on how discriminative and temporally stable their features are, and uses it to show that image-based encoders DINOv2 and SAM2.1 Hiera are better alignment targets than video or 3D encoders. Because those two encoders emphasize different frequency bands—DINOv2 low, SAM2 high—concatenating their normalized features creates a richer supervisory signal. With this fused alignment loss, a V-DiT at 400K steps beats the vanilla model at 1M steps, implying at least 2.5x faster convergence, and achieves the best FVD and FID at 1M as well. The method improves both unconditional and class-conditional generation, under both diffusion and flow objectives.

Core claim

Align4Gen is a training-time regularizer that aligns patch tokens of a video diffusion transformer with features extracted from pretrained image encoders via a lightweight MLP mapper and cosine distance. The key discovery is that a fused target, formed by concatenating normalized DINOv2 and SAM2.1 Hiera features, outperforms alignment to either encoder alone and accelerates convergence substantially: on UCF-101, the fusion model at 400K steps (FVD 206.73) surpasses the vanilla V-DiT at 1M steps (FVD 221.63), and at 1M steps it reaches FVD 187.46. The authors trace the fusion benefit to complementary frequency characteristics: DINOv2 captures low-frequency semantic structure and SAM2.1 Hiera

What carries the argument

The load-bearing object is a dual-encoder alignment loss. Patch tokens from a V-DiT block are projected through a shared MLP and matched by cosine distance to a target feature formed by concatenating L2-normalized DINOv2 and SAM2.1 Hiera patch embeddings (after per-encoder normalization). Two analyses motivate the choice: IICR (inter-cluster min distance over intra-cluster max std from K-means on per-frame features) ranks encoders by joint discriminability and temporal stability; a Fourier-based frequency gap measure shows DINOv2 emphasizes low frequencies while SAM2.1 Hiera emphasizes high frequencies. The concatenation is done after normalization so neither feature dominates, and the align

Load-bearing premise

The load-bearing premise is that the IICR score—how well an encoder's per-frame features form tight, well-separated clusters—correctly predicts how much that encoder will help as an alignment target during diffusion training; if that correlation breaks for other encoders or datasets, the selection of DINOv2 and SAM2.1 is no longer justified.

What would settle it

Train the same V-DiT with alignment to (a) DINOv2, (b) SAM2.1, and (c) a fused target of two encoders that both emphasize low frequencies (e.g., DINOv2 plus a second low-frequency encoder) under matched feature dimension. If (c) matches the DINOv2/SAM2 fusion in FVD, then the frequency-complementarity explanation is wrong and the gain comes from larger target dimension. Alternatively, if a new encoder with a higher IICR than DINOv2/SAM2 yields worse FVD than DINOv2/SAM2 when used alone, IICR is not a valid predictor.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Video diffusion training can be made at least 2.5x cheaper in steps by adding a frozen-encoder alignment loss, since 400K-step fusion matches or beats the 1M-step baseline.
  • IICR provides a cheap, training-free way to pre-screen vision encoders for use as alignment targets; the paper shows performance ordering across encoders (DINOv2/SAM2 > VideoMAE > DUSt3R) follows IICR ordering.
  • Complementarity in frequency (low vs high) of self-supervised encoders can be exploited by feature concatenation to yield a better supervisory signal than either encoder alone.
  • The alignment regularizer is orthogonal to the denoising objective (diffusion or flow) and improves both unconditional and class-conditional generation while increasing Inception Score, indicating no mode collapse.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • IICR could serve as a general selection criterion for auxiliary supervision sources in any generative model, not just video; a direct test would be applying it to image diffusion or text-to-image alignment.
  • The paper reports that alignment interfered with a text-to-image pretrained initialization, so a clean test of Align4Gen for text-to-video is to train from scratch, which the authors list as future work.
  • Fusion gain may not require exactly DINOv2/SAM2; any pair of encoders with separated frequency emphasis might work. A direct test would compare fusion of two low-frequency encoders against the DINOv2/SAM2 pair under matched total feature dimension.
  • The metric and method could extend to other modalities (audio, multi-view) where discriminability and temporal consistency trade off, though this is speculative.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Align4Gen, a training-time regularizer for video diffusion transformers (V-DiT) that aligns intermediate patch tokens with features from pre-trained image encoders. The authors introduce a metric, Intra-Inter Consistency Ratio (IICR), to assess encoder discriminability and temporal consistency, and use it to select DINOv2 and SAM2.1 Hiera as alignment targets. Because these two encoders exhibit complementary frequency biases (low-frequency vs. high-frequency, Fig. 3), the method concatenates their normalized features and minimizes a cosine-distance alignment loss (Eqs. 13–15) on top of the diffusion or flow-matching objective. Experiments on UCF-101 (class-conditional), SkyTimelapse, and FaceForensics (unconditional) show consistent FVD/FID improvements over a vanilla V-DiT and over single-encoder alignment, at both 400K and 1M iterations. The paper claims at least 2.5x faster convergence (Table 1) and reduced training cost. A supplementary limitation section reports that the method does not transfer to a text-to-video setting when initialized from pre-trained PixArt-α weights, motivating the paper's focus on from-scratch training.

Significance. If the claims hold, this is a useful and simple recipe for improving the sample efficiency and final quality of from-scratch video DiT training, with a systematic encoder-selection criterion (IICR) that others could reuse. The empirical study is fairly broad: it covers two training objectives (diffusion and flow), three datasets, two model scales, and ablations of alignment depth, encoder choice, and fusion versus separate MLPs. The frequency-complementarity explanation for fusion gains is plausible and supported by Fourier analysis. However, the headline 'reduced computational cost' claim is currently supported only by step counts, and the paper does not report the substantial per-step overhead of running two frozen large encoders. The limitation of the method to from-scratch training is relegated to the supplementary. These issues temper the practical significance and need to be addressed before the central claims can be accepted.

major comments (4)
  1. [§4 Results, Table 1, and Conclusion] The claim that Align4Gen is 'at least ×2.5 faster' and 'substantially reduc[es] the computational cost' is based solely on iteration counts (400K vs. 1M steps). Algorithm 1 (Suppl. B) shows that DINOv2 and SAM2.1 Hiera features are computed for every training video at every step; these are large frozen encoders, and the paper reports no wall-clock time, per-step overhead, FLOPs, or a matched-compute baseline. If the two extra forward passes cost, say, 50–100% of the baseline step, the step reduction would not translate into wall-clock savings. This is load-bearing because training-cost reduction is a headline contribution. Please report wall-clock training time or a compute-matched comparison.
  2. [§3, Eq. (7), and Table 4] The IICR metric is the basis for selecting DINOv2/SAM2.1, and the paper claims (Section 4) that 'performance metrics follow the order defined by our proposed IICR metric.' The only quantitative support is Table 4, a single run with no error bars. IICR uses K-means with a fixed K range, and the definitions of Dinter as the minimum centroid distance and Dintra as the maximum cluster standard deviation make the ratio sensitive to outliers and to K. The predictive claim would be much stronger with multiple K values, multiple seeds, and error bars on the resulting FVD values.
  3. [§4, Evaluation Metrics and Tables 1–2] The main quantitative claims rely entirely on the content-debiased FVD variant computed with VideoMAE features. The paper itself argues that VideoMAE features have low temporal consistency and low IICR, making them unsuitable as alignment targets; using them as the evaluation backbone is not contradictory, but it deserves more scrutiny. Since the paper also reports standard I3D-based FVD only in the supplementary (Table 7), the reader cannot assess whether the headline gains are robust to the choice of evaluation feature extractor. Please report both standard and content-debiased FVD for the main models, and include error bars or at least multiple evaluation seeds.
  4. [Suppl. E, Limitation] The supplementary states that in a text-to-video setting with text-to-image pre-initialization, the alignment loss 'interferes with models that already possess strong semantic priors' and leads to a decline in performance. This substantially narrows the scope of the claimed benefit: the method is validated for from-scratch training on small datasets, not for the more common large-scale or pretrained-initialized training regimes. The main text presents the method and conclusions without this caveat. The limitation should be stated prominently in the main paper and reflected in the claims.
minor comments (5)
  1. [§3, Eq. (13)] N is described as 'the total number of patch tokens' but the equation averages over a video of multiple frames; clarify whether N counts all frames or one frame. Also, the MLP is later described as shared across patches; state this in the main text, not only in the supplementary.
  2. [§3, Multi-Feature Fusion] The sentence 'This combination enables a more holistic representation enables capturing nuanced details beyond mere high-level semantics' contains a grammatical duplication. Also, define whether the normalization mentioned in Suppl. B (normalizing before concatenation) is part of the main method; the main text currently omits it.
  3. [§4, Implementation details] No value for the alignment loss weight γ is given for the main UCF-101/FaceForensics/SkyTimelapse experiments. The supplementary gives γ=0.5 for the text-to-video run only. Reporting the chosen γ and any sensitivity analysis would help reproducibility.
  4. [Tables and naming] The paper alternates between 'SAM2' and 'SAM2.1 Hiera'; standardize on the exact model name. Also, Table 7 in the supplementary lacks a comparison with the standard FVD of the main baseline; consider adding it.
  5. [General] No code or checkpoints are released. For a method whose contribution is largely empirical, releasing code would substantially improve reproducibility.

Circularity Check

0 steps flagged

No significant circularity: Align4Gen's encoder choice is motivated by independent metrics and validated by separate training runs, not forced by construction.

full rationale

Align4Gen's derivation chain is not circular. The encoder selection is guided by the proposed IICR metric (Eq. 7) and frequency analysis (Eq. 12), which are computed directly from frozen pretrained encoder features and K-means clustering; they are not fitted to downstream FVD/FID scores. The alignment loss (Eq. 13-14) supervises V-DiT features with fixed DINOv2/SAM2.1 Hiera targets, and its benefit is measured on held-out UCF-101, SkyTimelapse, and FaceForensics evaluations. Table 4's agreement with the IICR ranking is an empirical check, not a tautology: the IICR values and the FVD numbers come from separate procedures, and neither quantity is defined in terms of the other. The '×2.5 faster convergence' statement is arithmetic over two measured step-count curves (400K vs 1M), though it does not include wall-clock overhead from the frozen encoders; that is a correctness/efficiency concern, not circularity. Self-citations to Track4Gen and JOG3R appear only in related-work positioning and are not used to justify Align4Gen's design, and no uniqueness theorem or unverified prior claim is imported to rule out alternatives. The Suppl. E text-to-video limitation is a scope restriction and does not make the unconditional/class-conditional results definitional. No equation reduces to its own input, and no fitted parameter is renamed as a prediction.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The central claim rests on hand-chosen hyperparameters (gamma, depth, frequency cutoff) and on the validity of a new metric (IICR) that is not independently benchmarked. No new physical entities are introduced.

free parameters (4)
  • Alignment loss weight gamma = 0.5 (specified only for text-to-video in Suppl. E; unspecified for main experiments)
    Balances Lalign and Ldiff/RF; chosen by hand, no sensitivity analysis.
  • Feature alignment depth = 12 (V-DiT-XL) or 8 (V-DiT-L in Table 2)
    Hand-selected based on Table 3 ablations; affects which intermediate features are aligned.
  • High-frequency threshold 0.75r = 0.75
    Arbitrary cutoff defining the high-frequency region in Eq. 11 for the frequency analysis that motivates fusion; not varied or justified.
  • IICR cluster count K = varied (e.g., 1000 videos, K values)
    The IICR metric depends on K; the paper reports trends across K but does not derive a principled K or error bars.
axioms (4)
  • domain assumption Pretrained encoder features are meaningful supervision
    The method assumes aligning to DINOv2/SAM2 features improves generation; supported only by the paper's own experiments.
  • ad hoc to paper IICR predicts downstream generation quality
    Proposed in this paper; used to select encoders before validating with experiments; no external benchmark.
  • standard math Standard diffusion/flow matching training
    Assumes the DDPM and flow matching objectives (Eqs. 2, 4) as background.
  • domain assumption Cosine distance on normalized concatenated features is a suitable alignment objective
    Eq. 13 and Suppl. B; the choice is standard (REPA) but not independently motivated for video.

pith-pipeline@v1.3.0-alltime-deepseek · 18130 in / 12654 out tokens · 128446 ms · 2026-08-04T18:52:40.902976+00:00 · methodology

0 comments
read the original abstract

Video diffusion models have advanced rapidly in the recent years as a result of series of architectural innovations (e.g., diffusion transformers) and use of novel training objectives (e.g., flow matching). In contrast, less attention has been paid to improving the feature representation power of such models. In this work, we show that training video diffusion models can benefit from aligning the intermediate features of the video generator with feature representations of pre-trained vision encoders. We propose a new metric and conduct an in-depth analysis of various vision encoders to evaluate their discriminability and temporal consistency, thereby assessing their suitability for video feature alignment. Based on the analysis, we present Align4Gen which provides a novel multi-feature fusion and alignment method integrated into video diffusion model training. We evaluate Align4Gen both for unconditional and class-conditional video generation tasks and show that it results in improved video generation as quantified by various metrics. Full video results are available on our project page: https://align4gen.github.io/align4gen/

Figures

Figures reproduced from arXiv: 2509.09547 by Dohun Lee, Duygu Ceylan, Hyeonho Jeong, Jiwook Kim, Jong Chul Ye.

Figure 1
Figure 1. Figure 1 [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: PCA visualization of different vision encoder features. Pretrained image encoders—DINOv2 (Oquab et al. 2023) and SAM2 Hiera image encoder (Ravi et al. 2024)—yield feature representations that remain consistent across frames for both foreground objects and background. In contrast, video or 3D vision encoders such as VideoMAE (Tong et al. 2022) and DUSt3R (Wang et al. 2024) exhibit significant temporal incon… view at source ↗
Figure 3
Figure 3. Figure 3: Mean log differences between the lowest and highest frequency components. DINOv2 focuses more on low-frequency components, whereas SAM2.1 Hiera places greater emphasis on high-frequency regions. We also plot the same metric for the features obtained from a mid-block of a video diffusion transformer model as a reference. SAM2.1 Hiera as the primary sources for feature alignment in video diffusion. Frequency… view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative comparisons on UCF-101. Our method converges noticeably faster than the original model. Moreover, in addition to improved per-frame image quality, the X-T slice analysis reveals smoother frame transitions, highlighting enhanced visual fidelity and motion consistency in the V-DiT model. Here, the green line indicates the position of the X-T slice, and videos in the same row share the same green … view at source ↗
Figure 5
Figure 5. Figure 5: Qualitative comparison on FaceForensics. Our Fusion method preserves finer details and achieves higher fidelity than the baseline Method Depth Alignment FVD Spatial Temporal V-DiT 12 - - 379.90 + Ours (DINOv2) 12 ✓ - 311.14 + Ours (DINOv2) 8 ✓ - 313.18 + Ours (DINOv2) 12 - ✓ 315.41 [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: PCA visualization and tracking comparison videos of image and video encoders. Click each image to play the video in Acrobat Reader. FID Protocol. To compute FID for video generators, we follow the approach used in StyleGAN-V. Specifically, we generate 2,048 videos, each containing 16 frames, and use all frames for FID computation. For real statistics, similar to the FVD protocol, we sample 2,048 clips from… view at source ↗
Figure 7
Figure 7. Figure 7: (a) The V-DiT architecture, consisting of [PITH_FULL_IMAGE:figures/full_fig_p012_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Based on our PCA analysis of the SD3 transformer [PITH_FULL_IMAGE:figures/full_fig_p013_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: From left to right: input video, PCA visualization using InternVL features, IICR metric plot, and FVD scores. still exhibit reasonable performance. However, they intro￾duce a substantial computational overhead, running approx￾imately 5× slower than DINOv2. Due to time constraints, we trained InternVL-based models for up to 100K itera￾tions and compared them with baselines, as shown in [PITH_FULL_IMAGE:fig… view at source ↗
Figure 10
Figure 10. Figure 10: More qualitative results on the UCF-101 dataset for V-DiT versus V-DiT + Ours trained on 1M iterations, clearly [PITH_FULL_IMAGE:figures/full_fig_p015_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Qualitative results on the FaceForensics dataset for V-DiT versus V-DiT + Ours trained on 200K iterations, demon [PITH_FULL_IMAGE:figures/full_fig_p016_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Qualitative results on the SkyTimeLapse dataset for V-DiT versus V-DiT + Ours trained on 200K iterations. [PITH_FULL_IMAGE:figures/full_fig_p016_12.png] view at source ↗
Figure 14
Figure 14. Figure 14: Qualitative video result of Fig. 4 and Fig. 5. [PITH_FULL_IMAGE:figures/full_fig_p017_14.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

38 extracted references · 33 linked inside Pith

  1. [3]

    arXiv preprint arXiv:2502.04896

    Goku: Flow Based Video Generative Foundation Models. arXiv preprint arXiv:2502.04896. El Banani, M.; Raj, A.; Maninis, K.-K.; Kar, A.; Li, Y .; Ru- binstein, M.; Sun, D.; Guibas, L.; Johnson, J.; and Jampani, V

    Pith/arXiv arXiv

  2. [4]

    Motivated by this trend, we also analyzed the vision encoder used in a state-of-the-art MLLM to better understand its po- tential for video generation tasks

    Analysis of MLLM Feature Representation Recently, multi-modal large language models (MLLMs) have emerged as a rapidly growing research direction, with vision encoders playing a crucial role in their performance. Motivated by this trend, we also analyzed the vision encoder used in a state-of-the-art MLLM to better understand its po- tential for video gener...

    2023

  3. [5]

    arXiv preprint arxiv:2307.10373

    TokenFlow: Consistent Diffusion Features for Consistent Video Editing. arXiv preprint arxiv:2307.10373. Guo, Y .; Yang, C.; Rao, A.; Liang, Z.; Wang, Y .; Qiao, Y .; Agrawala, M.; Lin, D.; and Dai, B

    Pith/arXiv arXiv

  4. [6]

    arXiv preprint arXiv:2307.04725

    Animatediff: Ani- mate your personalized text-to-image diffusion models with- out specific tuning. arXiv preprint arXiv:2307.04725. HaCohen, Y .; Chiprut, N.; Brazowski, B.; Shalem, D.; Moshe, D.; Richardson, E.; Levin, E.; Shiran, G.; Zabari, N.; Gordon, O.; et al

    Pith/arXiv arXiv

  5. [7]

    arXiv preprint arXiv:2501.00103

    Ltx-video: Realtime video la- tent diffusion. arXiv preprint arXiv:2501.00103. He, K.; Chen, X.; Xie, S.; Li, Y .; Doll´ar, P.; and Girshick, R

    Pith/arXiv arXiv

  6. [10]

    arXiv preprint arXiv:2501.01409

    On Unifying Video Generation and Camera Pose Estimation. arXiv preprint arXiv:2501.01409. Jeong, H.; Huang, C.-H. P.; Ye, J. C.; Mitra, N.; and Cey- lan, D

    Pith/arXiv arXiv

  7. [11]

    arXiv preprint arXiv:2412.06016

    Track4Gen: Teaching Video Diffusion Models to Track Points Improves Video Generation. arXiv preprint arXiv:2412.06016. Ju, X.; Gao, Y .; Zhang, Z.; Yuan, Z.; Wang, X.; Zeng, A.; Xiong, Y .; Xu, Q.; and Shan, Y

    Pith/arXiv arXiv

  8. [12]

    arXiv preprint arXiv:2501.12218

    Exploring Temporally-Aware Features for Point Tracking. arXiv preprint arXiv:2501.12218. Kong, W.; Tian, Q.; Zhang, Z.; Min, R.; Dai, Z.; Zhou, J.; Xiong, J.; Li, X.; Wu, B.; Zhang, J.; et al

    Pith/arXiv arXiv

  9. [13]

    arXiv preprint arXiv:2412.03603

    Hunyuan- video: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03603. Lipman, Y .; Chen, R. T.; Ben-Hamu, H.; Nickel, M.; and Le, M

    Pith/arXiv arXiv

  10. [15]

    arXiv preprint arXiv:2209.03003

    Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003. Luo, G.; Dunlap, L.; Park, D. H.; Holynski, A.; and Darrell, T

    Pith/arXiv arXiv

  11. [16]

    arXiv:2502.10248

    Step-Video-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Model. arXiv:2502.10248. Ma, X.; Wang, Y .; Jia, G.; Chen, X.; Liu, Z.; Li, Y .-F.; Chen, C.; and Qiao, Y

    Pith/arXiv arXiv

  12. [17]

    arXiv preprint arXiv:2401.03048

    Latte: Latent Diffusion Transformer for Video Generation. arXiv preprint arXiv:2401.03048. Nan, K.; Xie, R.; Zhou, P.; Fan, T.; Yang, Z.; Chen, Z.; Li, X.; Yang, J.; and Tai, Y

    Pith/arXiv arXiv

  13. [18]

    Oquab, M.; Darcet, T.; Moutakanni, T.; V o, H

    Openvid-1m: A large- scale high-quality dataset for text-to-video generation.arXiv preprint arXiv:2407.02371. Oquab, M.; Darcet, T.; Moutakanni, T.; V o, H. V .; Szafraniec, M.; Khalidov, V .; Fernandez, P.; Haziza, D.; Massa, F.; El-Nouby, A.; Howes, R.; Huang, P.-Y .; Xu, H.; Sharma, V .; Li, S.-W.; Galuba, W.; Rabbat, M.; Assran, M.; Ballas, N.; Synnae...

    Pith/arXiv arXiv

  14. [19]

    arXiv preprint arXiv:2307.01952

    Sdxl: Im- proving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952. Polyak, A.; Zohar, A.; Brown, A.; Tjandra, A.; Sinha, A.; Lee, A.; Vyas, A.; Shi, B.; Ma, C.-Y .; Chuang, C.-Y .; Yan, D.; Choudhary, D.; Wang, D.; Sethi, G.; Pang, G.; Ma, H.; Misra, I.; Hou, J.; Wang, J.; Jagadeesh, K.; Li, K.; Zhang, L.; Sing...

    Pith/arXiv arXiv

  15. [20]

    arXiv:2410.13720

    Movie Gen: A Cast of Media Foundation Models. arXiv:2410.13720. Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; Krueger, G.; and Sutskever, I

    Pith/arXiv arXiv

  16. [21]

    arXiv preprint arXiv:2408.00714

    SAM 2: Segment Anything in Images and Videos. arXiv preprint arXiv:2408.00714. Rombach, R.; Blattmann, A.; Lorenz, D.; Esser, P.; and Om- mer, B

    Pith/arXiv arXiv

  17. [22]

    R¨ossler, A.; Cozzolino, D.; Verdoliva, L.; Riess, C.; Thies, J.; and Nießner, M

    arXiv preprint arXiv:1803.09179. R¨ossler, A.; Cozzolino, D.; Verdoliva, L.; Riess, C.; Thies, J.; and Nießner, M

    Pith/arXiv arXiv

  18. [23]

    arXiv preprint arXiv:2209.14792

    Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792. Skorokhodov, I.; Tulyakov, S.; and Elhoseiny, M

    Pith/arXiv arXiv

  19. [26]

    arXiv:2502.06755

    Sparse Autoencoders for Scientifically Rigorous Interpreta- tion of Vision Models. arXiv:2502.06755. Tang, L.; Jia, M.; Wang, Q.; Phoo, C. P.; and Hariharan, B

    arXiv

  20. [29]

    arXiv:2312.14132

    DUSt3R: Geometric 3D Vision Made Easy. arXiv:2312.14132. Wang, Y .; He, Y .; Li, Y .; Li, K.; Yu, J.; Ma, X.; Li, X.; Chen, G.; Chen, X.; Wang, Y .; et al

    Pith/arXiv arXiv

  21. [30]

    arXiv preprint arXiv:2307.06942

    Internvid: A large-scale video-text dataset for multimodal understanding and gener- ation. arXiv preprint arXiv:2307.06942. Weinzaepfel, P.; Leroy, V .; Lucas, T.; Br ´egier, R.; Cabon, Y .; Arora, V .; Antsfeld, L.; Chidlovskii, B.; Csurka, G.; and Revaud, J

    Pith/arXiv arXiv

  22. [31]

    arXiv preprint arXiv:2408.06072

    Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072. Yu, S.; Kwak, S.; Jang, H.; Jeong, J.; Huang, J.; Shin, J.; and Xie, S

    Pith/arXiv arXiv

  23. [32]

    arXiv preprint arXiv:2410.06940

    Representation alignment for genera- tion: Training diffusion transformers is easier than you think. arXiv preprint arXiv:2410.06940. Yu, S.; Sohn, K.; Kim, S.; and Shin, J

    Pith/arXiv arXiv

  24. [33]

    arXiv preprint arxiv:2410.03825

    MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motion. arXiv preprint arxiv:2410.03825. Zhang, J.; Herrmann, C.; Hur, J.; Polania Cabrera, L.; Jam- pani, V .; Sun, D.; and Yang, M.-H

    Pith/arXiv arXiv

  25. [34]

    arXiv preprint arXiv:2410.15458

    Allegro: Open the black box of commercial-level video generation model. arXiv preprint arXiv:2410.15458. This supplementary material is organized as follows: (A) Evaluation Metrics (B) Additional Experimental Details (C) V-DiT Architecture (D) Additional Feature Analysis (E) Limitation (F) Comparison to Previous Methods (G) Additional Qualitative Results ...

    Pith/arXiv arXiv 2024

  26. [35]

    is a widely used action recognition dataset that contains 13,320 videos spanning 101 action categories. For our experiments, we use only the training split for model training, ensuring Figure 7: (a) The V-DiT architecture, consisting of N trans- former blocks, each comprising one spatial block and one temporal block. (b) The architecture of a single trans...

    2023

  27. [36]

    We follow the same protocol as SkyTimeLapse, using the training split for both model training and metric evaluations

    is a dataset designed for facial manipulation detection, contain- ing both real and manipulated face videos. We follow the same protocol as SkyTimeLapse, using the training split for both model training and metric evaluations. C V-DiT Architecture V-DiT constructs the latent space representation of a video by processing frames individually using the V AE ...

    2022

  28. [37]

    and Chrono PointTrack (Kim et al. 2025). Analysis of Generative Model Feature Representation (SD3) We conducted a PCA analysis of features extracted from the transformer blocks of SD3 at different layer depths. The re- sults showed that as the layer depth increased, the extracted information became more detailed and well-defined. Based on the PCA results ...

    2025

  29. [2012]

    arXiv:1212.0402

    UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild. arXiv:1212.0402. Stevens, S.; Chao, W.-L.; Berger-Wolf, T.; and Su, Y

    Pith/arXiv arXiv

  30. [2015]

    arXiv:1412.0767

    Learning Spatiotemporal Features with 3D Con- volutional Networks. arXiv:1412.0767. Unterthiner, T.; van Steenkiste, S.; Kurach, K.; Marinier, R.; Michalski, M.; and Gelly, S

    Pith/arXiv arXiv

  31. [2018]

    arXiv:1706.08500

    GANs Trained by a Two Time- Scale Update Rule Converge to a Local Nash Equilibrium. arXiv:1706.08500. Ho, J.; Jain, A.; and Abbeel, P

    Pith/arXiv arXiv

  32. [2019]

    arXiv:1812.01717

    Towards Accurate Generative Models of Video: A New Metric & Challenges. arXiv:1812.01717. Wang, Q.; Eldesokey, A.; Mendiratta, M.; Zhan, F.; Ko- rtylewski, A.; Theobalt, C.; and Wonka, P

    Pith/arXiv arXiv

  33. [2020]

    arXiv:2010.02502

    Denoising Diffu- sion Implicit Models. arXiv:2010.02502. Soomro, K.; Zamir, A. R.; and Shah, M

    Pith/arXiv arXiv 2010

  34. [2021]

    arXiv:2111.06377

    Masked Autoencoders Are Scalable Vision Learners. arXiv:2111.06377. Heusel, M.; Ramsauer, H.; Unterthiner, T.; Nessler, B.; and Hochreiter, S

    Pith/arXiv arXiv

  35. [2022]

    arXiv preprint arXiv:2210.02747

    Flow matching for generative modeling. arXiv preprint arXiv:2210.02747. Liu, X.; Gong, C.; and Liu, Q

    Pith/arXiv arXiv

  36. [2023]

    arXiv:2310.00426

    PixArt- α: Fast Training of Diffusion Transformer for Photorealistic Text- to-Image Synthesis. arXiv:2310.00426. Chen, R. T.; Rubanova, Y .; Bettencourt, J.; and Duvenaud, D. K

    Pith/arXiv arXiv

  37. [2024]

    arXiv:2403.03206

    Scaling Rectified Flow Transformers for High-Resolution Image Synthesis. arXiv:2403.03206. Ge, S.; Mahapatra, A.; Parmar, G.; Zhu, J.-Y .; and Huang, J.-B

    Pith/arXiv arXiv

  38. [2025]

    arXiv preprint arXiv:2502.02492

    Video- JAM: Joint Appearance-Motion Representations for En- hanced Motion Generation in Video Models. arXiv preprint arXiv:2502.02492. Chen, J.; Yu, J.; Ge, C.; Yao, L.; Xie, E.; Wu, Y .; Wang, Z.; Kwok, J.; Luo, P.; Lu, H.; and Li, Z

    Pith/arXiv arXiv