REVIEW 10 cited by
MotionDirector: Motion Customization of Text-to-Video Diffusion Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Large-scale pre-trained diffusion models have exhibited remarkable capabilities in diverse video generations. Given a set of video clips of the same motion concept, the task of Motion Customization is to adapt existing text-to-video diffusion models to generate videos with this motion. For example, generating a video with a car moving in a prescribed manner under specific camera movements to make a movie, or a video illustrating how a bear would lift weights to inspire creators. Adaptation methods have been developed for customizing appearance like subject or style, yet unexplored for motion. It is straightforward to extend mainstream adaption methods for motion customization, including full model tuning, parameter-efficient tuning of additional layers, and Low-Rank Adaptions (LoRAs). However, the motion concept learned by these methods is often coupled with the limited appearances in the training videos, making it difficult to generalize the customized motion to other appearances. To overcome this challenge, we propose MotionDirector, with a dual-path LoRAs architecture to decouple the learning of appearance and motion. Further, we design a novel appearance-debiased temporal loss to mitigate the influence of appearance on the temporal training objective. Experimental results show the proposed method can generate videos of diverse appearances for the customized motions. Our method also supports various downstream applications, such as the mixing of different videos with their appearance and motion respectively, and animating a single image with customized motions. Our code and model weights will be released.
Forward citations
Cited by 10 Pith papers
-
SketchAnimator: Animate Sketch via Motion Customization of Text-to-Video Diffusion Models
A three-stage method (appearance LoRA, motion LoRA, SDS stroke optimization) animates a user sketch with the motion of a reference video in a one-shot setting.
-
Scaling Up Audio-Synchronized Visual Animation: An Efficient Training Paradigm
An audio-conditioned video animation model is pretrained on noisy auto-curated videos and fine-tuned on a few clean examples, achieving top synchronization scores on a new 48-class benchmark with only 1.9% additional ...
-
AffordanceSAM: Segment Anything Once More in Affordance Grounding
Adapting EVF-SAM with learnable affordance queries and a coarse-to-fine dataset yields strong affordance grounding on AGD20K, with caveats about test-set tuning.
-
Separate Motion from Appearance: Customizing Motion via Customizing Text-to-Video Diffusion Models
A temporal attention purification and skip-connection rerouting method that separates motion learning from appearance learning in text-to-video diffusion model customization.
-
Llama Learns to Direct: DirectorLLM for Human-Centric Video Generation
A fine-tuned Llama 3 LLM writes discrete human pose tokens from a text prompt, and a pose-conditioned diffusion video renderer turns them into videos, improving human motion fidelity.
-
Beyond Generation: Unlocking Universal Editing via Self-Supervised Fine-Tuning
UES adds a self-supervised video condition to text-to-video diffusion models, enabling them to edit videos from delta prompts without paired supervision.
-
MoTrans: Customized Motion Transfer with Text-driven Video Diffusion Models
MoTrans transfers specific motions from reference videos to new subjects using a two-stage fine-tuning scheme with recaptioned prompts, appearance injection, and a motion-specific verb embedding.
-
Trajectory Attention for Fine-grained Video Motion Control
An auxiliary trajectory attention branch, added to temporal attention in video diffusion models, improves camera motion control precision while preserving generation quality.
-
PhysMotion: Physics-Grounded Dynamics From a Single Image
PhysMotion generates physically plausible videos from a single image by simulating 3D object motion with a material point method, then enhancing the rendering with a diffusion model.
-
PhysAnimator: Physics-Guided Generative Cartoon Animation
PhysAnimator combines 2D deformable-body physics simulation with a sketch-guided video diffusion model to animate static anime illustrations with controllable, physically plausible motion.
Discussion (0). Continue with ORCID to comment.