REVIEW 23 cited by
Latent-Shift: Latent Diffusion with Temporal Shift for Efficient Text-to-Video Generation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
We propose Latent-Shift -- an efficient text-to-video generation method based on a pretrained text-to-image generation model that consists of an autoencoder and a U-Net diffusion model. Learning a video diffusion model in the latent space is much more efficient than in the pixel space. The latter is often limited to first generating a low-resolution video followed by a sequence of frame interpolation and super-resolution models, which makes the entire pipeline very complex and computationally expensive. To extend a U-Net from image generation to video generation, prior work proposes to add additional modules like 1D temporal convolution and/or temporal attention layers. In contrast, we propose a parameter-free temporal shift module that can leverage the spatial U-Net as is for video generation. We achieve this by shifting two portions of the feature map channels forward and backward along the temporal dimension. The shifted features of the current frame thus receive the features from the previous and the subsequent frames, enabling motion learning without additional parameters. We show that Latent-Shift achieves comparable or better results while being significantly more efficient. Moreover, Latent-Shift can generate images despite being finetuned for T2V generation.
Forward citations
Cited by 23 Pith papers
-
VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Models
Adding a joint appearance-motion prediction objective and an inner-guidance mechanism to a diffusion transformer video model substantially improves motion coherence, motion smoothness, and human preference for motion realism.
-
OmniCustom: Sync Audio-Video Customization Via Joint Audio-Video Generation Model
A zero-shot model that generates a video of a reference face speaking user-chosen text with a reference voice timbre.
-
Emergent Temporal Correspondences from Video Diffusion Transformers
Video diffusion transformers encode temporal correspondences primarily in query-key similarities of a few specific attention layers, which can be extracted for zero-shot point tracking and used for training-free motio...
-
FlowMo: Variance-Based Flow Guidance for Coherent Motion in Video Generation
FlowMo reduces temporal artifacts in video generation by guiding the denoising process to lower the maximum patch-wise variance of consecutive-frame differences in the latent space.
-
Pursuing Temporal-Consistent Video Virtual Try-On via Dynamic Pose Interaction
DPIDM, a diffusion model with pose-aware spatial and temporal attention plus a temporal attention loss, reports state-of-the-art video virtual try-on and cuts VFID on VVT from 1.280 to 0.506.
-
DualReal: Adaptive Joint Training for Lossless Identity-Motion Fusion in Video Customization
DualReal jointly trains identity and motion adapters for video customization, reporting a 21.7% CLIP-I and 31.8% DINO-I improvement over existing methods.
-
Separate Motion from Appearance: Customizing Motion via Customizing Text-to-Video Diffusion Models
A temporal attention purification and skip-connection rerouting method that separates motion learning from appearance learning in text-to-video diffusion model customization.
-
Ouroboros-Diffusion: Exploring Consistent Content Generation in Tuning-free Long Video Diffusion
Ouroboros-Diffusion improves long video consistency by combining low-frequency tail noise, subject-aware cross-frame attention, and self-recurrent gradient guidance in a tuning-free FIFO diffusion queue.
-
Through-The-Mask: Mask-based Motion Trajectories for Image-to-Video Generation
A mask-based motion trajectory representation plus masked attention improves multi-object image-to-video generation.
-
MotiF: Making Text Count in Image Animation with Motion Focal Loss
A motion-weighted training loss (MotiF) improves text alignment and object motion in text-image-to-video generation, winning 72% of human-preference comparisons against nine baselines on a new benchmark.
-
AKiRa: Augmentation Kit on Rays for optical video generation
AKiRa adds an augmentation scheme and an aperture map to a camera-controlled video diffusion model, enabling control of focal length, distortion, and bokeh.
-
TIV-Diffusion: Towards Object-Centric Movement for Text-driven Image to Video Generation
TIV-Diffusion adds object-centric slot alignment to a diffusion-based image-to-video generator and reports improved alignment and temporal-consistency metrics on MNIST, CATER, and Bridge datasets.
-
Factorized Video Autoencoders for Efficient Generative Modelling
A four-plane factorized video latent space cuts the diffusion model sequence length from 1280 to 672 tokens, achieving FVD 38 versus 39 for the volumetric W.A.L.T. baseline on UCF-101 at roughly half the compute.
-
Animate-X++: Universal Character Image Animation with Dynamic Backgrounds
Animate-X++ turns cartoon images into pose-driven animations with text-controlled moving backgrounds, claiming state-of-the-art results on a new synthetic anthropomorphic benchmark.
-
FinePhys: Fine-grained Human Action Generation by Explicitly Incorporating Physical Laws for Effective Skeletal Guidance
FinePhys adds a learned PhysNet that re-estimates 3D skeletons through an Euler-Lagrange parameterization and uses the fused skeletons as 2D heatmap guidance for a latent diffusion model, reporting improved generation...
-
DAPE: Dual-Stage Parameter-Efficient Fine-Tuning for Consistent Video Editing with Diffusion Models
A dual-stage fine-tuning recipe, norm tuning followed by a visual adapter, improves temporal consistency and text alignment in one-shot video editing, with a new 232-video benchmark.
-
"See What I Imagine, Imagine What I See": Human-AI Co-Creation System for 360$^\circ$ Panoramic Video Generation in VR
A proof-of-concept VR workflow that combines speech-based prompt refinement, egocentric focal adjustment, and segment-wise iteration to co-create 360-degree panoramic videos with AI.
-
RepVideo: Rethinking Cross-Layer Representation for Video Generation
A feature cache and gating mechanism that aggregates neighboring layer outputs in a video diffusion transformer improves temporal coherence and spatial accuracy.
-
Physical Informed Driving World Model
DrivePhysica adds coordinate alignment, 3D instance flow, and box-coordinate guidance to a diffusion world model, achieving state-of-the-art FID/FVD on nuScenes and improving StreamPETR NDS by 3.6 points when mixed wi...
-
Latent-Reframe: Enabling Camera Control for Video Diffusion Model without Training
Latent-Reframe steers a pre-trained video diffusion model along a target camera trajectory by reframing halfway-denoised latents with time-aware 3D point clouds and then inpainting the resulting gaps, all without fine-tuning.
-
HuViDPO:Enhancing Video Generation through Direct Preference Optimization for Human-Centric Alignment
HuViDPO claims the first DPO-based alignment for text-to-video generation, but its loss reduces to the known DPO-SDXL objective and the evaluation is not reproducible.
-
Video Latent Flow Matching: Optimal Polynomial Projections for Video Interpolation and Extrapolation
VLFM models video latent patches as a HiPPO-LegS polynomial flow and trains a flow matching model to generate frames, claiming bounded interpolation and extrapolation error.
-
RichSpace: Enriching Text-to-Video Prompt Space via Text Embedding Interpolation
Picking an interpolated text embedding with cosine similarity to a perpendicular foot can make a text-to-video model generate composite-feature videos that direct prompts miss.
Discussion (0). Continue with ORCID to comment.