REVIEW 6 cited by
VDT: General-purpose Video Diffusion Transformers via Mask Modeling
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
This work introduces Video Diffusion Transformer (VDT), which pioneers the use of transformers in diffusion-based video generation. It features transformer blocks with modularized temporal and spatial attention modules to leverage the rich spatial-temporal representation inherited in transformers. We also propose a unified spatial-temporal mask modeling mechanism, seamlessly integrated with the model, to cater to diverse video generation scenarios. VDT offers several appealing benefits. 1) It excels at capturing temporal dependencies to produce temporally consistent video frames and even simulate the physics and dynamics of 3D objects over time. 2) It facilitates flexible conditioning information, \eg, simple concatenation in the token space, effectively unifying different token lengths and modalities. 3) Pairing with our proposed spatial-temporal mask modeling mechanism, it becomes a general-purpose video diffuser for harnessing a range of tasks, including unconditional generation, video prediction, interpolation, animation, and completion, etc. Extensive experiments on these tasks spanning various scenarios, including autonomous driving, natural weather, human action, and physics-based simulation, demonstrate the effectiveness of VDT. Additionally, we present comprehensive studies on how \model handles conditioning information with the mask modeling mechanism, which we believe will benefit future research and advance the field. Project page: https:VDT-2023.github.io
Forward citations
Cited by 6 Pith papers
-
Event Stream based Multi-Modal Video Anomaly Detection: A Benchmark Dataset and Algorithms
E-VAD plus the real visible–event TJUTCM Pha dataset improve weakly supervised video anomaly detection by contrastively aligning event, video, and text features and adaptively fusing them.
-
CDP: Towards Robust Autoregressive Visuomotor Policy Learning via Causal Diffusion
Causal Diffusion Policy adds historical action conditioning and attention cache sharing to diffusion-based robot policies, improving success rates on most tested manipulation tasks under degraded observations.
-
Programmatic Video Prediction Using Large Language Models
ProgGen uses language-model-written programs for perception, dynamics, and rendering to predict future video frames from about ten training examples, beating large diffusion baselines on two synthetic benchmarks.
-
Occlusion-Aware Temporally Consistent Amodal Completion for 3D Human-Object Interaction Reconstruction
A temporal feature warping and attention fusion module for amodal completion improves occlusion handling and temporal stability in monocular HOI videos, and the completed frames support 3D Gaussian Splatting reconstruction.
-
SlotPi: Physics-informed Object-centric Reasoning Models
SlotPi combines a learned Hamiltonian energy module with spatiotemporal attention to improve object-centric video prediction and visual question answering on several datasets.
-
A Diffusion-Driven Temporal Super-Resolution and Spatial Consistency Enhancement Framework for 4D MRI imaging
TSSC-Net uses a diffusion model conditioned on start and end frames to generate 6x more MRI time frames, then a tri-directional Mamba network to fix spatial inconsistencies.
Discussion (0). Sign in to comment.