REVIEW 17 cited by
MIMO: Controllable Character Video Synthesis with Spatial Decomposed Modeling
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Character video synthesis aims to produce realistic videos of animatable characters within lifelike scenes. As a fundamental problem in the computer vision and graphics community, 3D works typically require multi-view captures for per-case training, which severely limits their applicability of modeling arbitrary characters in a short time. Recent 2D methods break this limitation via pre-trained diffusion models, but they struggle for pose generality and scene interaction. To this end, we propose MIMO, a novel framework which can not only synthesize character videos with controllable attributes (i.e., character, motion and scene) provided by simple user inputs, but also simultaneously achieve advanced scalability to arbitrary characters, generality to novel 3D motions, and applicability to interactive real-world scenes in a unified framework. The core idea is to encode the 2D video to compact spatial codes, considering the inherent 3D nature of video occurrence. Concretely, we lift the 2D frame pixels into 3D using monocular depth estimators, and decompose the video clip to three spatial components (i.e., main human, underlying scene, and floating occlusion) in hierarchical layers based on the 3D depth. These components are further encoded to canonical identity code, structured motion code and full scene code, which are utilized as control signals of synthesis process. The design of spatial decomposed modeling enables flexible user control, complex motion expression, as well as 3D-aware synthesis for scene interactions. Experimental results demonstrate effectiveness and robustness of the proposed method.
Forward citations
Cited by 17 Pith papers
-
Perception-as-Control: Fine-grained Controllable Image Animation with 3D-aware Motion Representation
A diffusion-based image animation method that uses rendered colored spheres and a checkerboard world envelope as 3D-aware motion control signals, enabling joint camera and object motion control with fine granularity.
-
MultiAnimate: A Unified Framework for Controllable Multi-Character Animation
A diffusion-based framework that animates multiple characters in one scene from separate reference images and pose sequences while preserving each character's identity.
-
Precise Action-to-Video Generation Through Visual Action Prompts
Skeleton-based visual action prompts give precise, cross-domain action control for video generation of human and robot interactions.
-
PERSONA: Personalized Whole-Body 3D Avatar with Pose-Driven Deformations from a Single Image
PERSONA creates a personalized 3D avatar from one image by using diffusion-generated pose-rich videos to train a 3D Gaussian avatar with balanced sampling and geometry-weighted optimization.
-
Vid-CamEdit: Video Camera Trajectory Editing with Generative Rendering from Estimated Geometry
Vid-CamEdit re-synthesizes monocular videos along user-defined camera paths by conditioning a video diffusion model on 2D flows derived from estimated 3D geometry, without training on multi-view video data.
-
DreamActor-H1: High-Fidelity Human-Product Demonstration Video Generation via Motion-designed Diffusion Transformers
A diffusion transformer model generates human-product demonstration videos from paired human and product images while preserving both identities through masked cross-attention and motion template guidance.
-
HunyuanVideo-HOMA: Generic Human-Object Interaction in Multimodal Driven Human Animation
HunyuanVideo-HOMA generates human-object interaction videos from weak, sparse inputs: one arm pose, an object center dot, a human photo, and an object photo.
-
DreamDance: Animating Character Art via Inpainting Stable Gaussian Worlds
DreamDance animates a single character artwork by reconstructing its background as a 3D Gaussian scene and then inpainting the animated character into the rendered video.
-
AniCrafter: Customizing Realistic Human-Centric Animation via Avatar-Background Conditioning in Video Diffusion Models
A diffusion model animates a character into arbitrary dynamic backgrounds by conditioning on a rendered 3D-avatar video, reframing open-domain animation as a restoration problem.
-
AnyCharV: Bootstrap Controllable Character Video Generation with Fine-to-Coarse Guidance
AnyCharV is a two-stage diffusion method that places a reference character into a target video scene using pose and mask guidance, with a self-boosting stage that improves identity preservation.
-
ManiVideo: Generating Hand-Object Manipulation Video with Dexterous and Generalizable Grasping
ManiVideo generates bimanual hand-object manipulation videos conditioned on 3D motion sequences, using a multi-layer occlusion representation and Objaverse-based training to improve 3D consistency and object generalization.
-
AniGS: Animatable Gaussian Avatar from a Single Image with Inconsistent Gaussian Reconstruction
AniGS produces an animatable 3D avatar from a single image by synthesizing multi-view canonical images and normals with a video diffusion model and reconstructing them via 4D Gaussian Splatting.
-
DreamDance: Animating Human Images by Enriching 3D Geometry Cues from 2D Poses
A two-stage diffusion pipeline that enriches 2D pose guidance with generated depth and normal maps to achieve state-of-the-art human image animation.
-
PersonaCraft: Personalized and Controllable Full-Body Multi-Human Scene Generation Using Occlusion-Aware 3D-Conditioned Diffusion
PersonaCraft adds SMPLx depth and normal conditioning, occlusion boundary enhancement, and occlusion-aware classifier-free guidance to diffusion models, enabling controllable multi-person images that preserve both fac...
-
Drive Any Mesh: 4D Latent Diffusion for Mesh Deformation from Video
A video-conditioned latent diffusion model generates mesh vertex trajectories that deform an input 3D asset into render-ready 4D animations.
-
OmniV2V: Versatile Video Generation and Editing via Dynamic Content Manipulation
OmniV2V is one diffusion-transformer model that performs eight video generation and editing tasks by combining mask, pose, image, and text-instruction conditions.
-
Animate Anyone 2: High-Fidelity Character Image Animation with Environment Affordance
A diffusion-based character animation method that conditions on environment, object, and depth signals to produce videos where characters interact naturally with their surroundings.
Discussion (0). Continue with ORCID to comment.