Pith. sign in

REVIEW 11 cited by

MuseTalk: Real-Time High-Fidelity Video Dubbing via Spatio-Temporal Sampling

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.10122 v3 pith:QDOUVJUN submitted 2024-10-14 cs.CV

Yue Zhang , Zhizhou Zhong , Minhao Liu , Zhaokang Chen , Bin Wu , Yubin Zeng , Chao Zhan , Yingjie He
show 2 more authors
Junxin Huang Wenjiang Zhou
This is my paper
classification cs.CV
keywords musetalksamplingwhilelip-syncreal-timeaccuracyachievingaudio-visual
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Real-time video dubbing that preserves identity consistency while achieving accurate lip synchronization remains a critical challenge. Existing approaches face a trilemma: diffusion-based methods achieve high visual fidelity but suffer from prohibitive computational costs, while GAN-based solutions sacrifice lip-sync accuracy or dental details for real-time performance. We present MuseTalk, a novel two-stage training framework that resolves this trade-off through latent space optimization and spatio-temporal data sampling strategy. Our key innovations include: (1) During the Facial Abstract Pretraining stage, we propose Informative Frame Sampling to temporally align reference-source pose pairs, eliminating redundant feature interference while preserving identity cues. (2) In the Lip-Sync Adversarial Finetuning stage, we employ Dynamic Margin Sampling to spatially select the most suitable lip-movement-promoting regions, balancing audio-visual synchronization and dental clarity. (3) MuseTalk establishes an effective audio-visual feature fusion framework in the latent space, delivering 30 FPS output at 256*256 resolution on an NVIDIA V100 GPU. Extensive experiments demonstrate that MuseTalk outperforms state-of-the-art methods in visual fidelity while achieving comparable lip-sync accuracy. %The codes and models will be made publicly available upon acceptance. The code is made available at \href{https://github.com/TMElyralab/MuseTalk}{https://github.com/TMElyralab/MuseTalk}

Discussion (0). Sign in to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Geometry-guided Emotion Modulation for Controllable and Photorealistic Emotional Talking Face Generation

    cs.CV 2026-08 conditional novelty 6.0 of 10

    A diffusion-based talking-face generator uses 3D blendshape coefficients to continuously control the emotion intensity of generated facial expressions.

  2. LeapTalk: Breaking the Latency-Quality Trade-off in Talking Head Generation

    cs.CV 2026-07 conditional novelty 6.0 of 10

    LeapTalk distills a multi-step diffusion teacher into a one-step Brownian-bridge student and reports stable streaming talking-head generation at up to 200 FPS.

  3. Fre-Res: Frequency-Residual Video Token Compression for Efficient Video MLLMs

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    Temporal 1D-DCT of inter-frame latent residuals, fused into sparse spatial anchors by a cross-attention absorber, compresses video-MLLM visual tokens 2–22× while keeping accuracy within about 0.3–3 points of full-toke...

  4. MIND-V: Hierarchical World Model for Long-Horizon Robotic Manipulation with RL-based Physical Alignment

    cs.RO 2025-12 conditional novelty 6.0 of 10

    MIND-V generates long-horizon robot manipulation videos by decomposing instructions into sub-tasks with a VLM, encoding plans into a structured bridge, and fine-tuning a video diffusion model with a V-JEPA2-based phys...

  5. Who is a Better Talker: Subjective and Objective Quality Assessment for AI-Generated Talking Heads

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A new large dataset and the FSCD model improve automated quality scoring of AI-generated talking-head videos, beating 15 baselines in correlation with human ratings.

  6. JOLT3D: Joint Learning of Talking Heads and 3DMM Parameters with Application to Lip-Sync

    cs.CV 2025-07 conditional novelty 6.0 of 10

    JOLT3D jointly trains a 3DMM reconstruction network with a talking head generator, then uses FACS mouth blendshapes from a diffusion model to lip-sync videos while preserving the original chin contour.

  7. MS-DETR: Towards Effective Video Moment Retrieval and Highlight Detection by Joint Motion-Semantic Learning

    cs.CV 2025-07 conditional novelty 6.0 of 10

    MS-DETR improves moment retrieval and highlight detection by disentangling motion and semantic video features, sharing task information between the two tasks, and training on generated auxiliary captions.

  8. Fine-Grained Zero-Shot Object Detection

    cs.CV 2025-07 conditional novelty 6.0 of 10

    The authors define fine-grained zero-shot object detection, build a 1,432-species bird benchmark (FGZSD-Birds), and show their hierarchical MSHC detector outperforms prior ZSD models on that benchmark.

  9. FlashRT: Agent Harness for Guiding Agents to Deploy Real-Time Multimodal Applications

    cs.LG 2026-07 conditional novelty 5.0 of 10

    FlashRT's agent harness converts reference multimodal pipelines into optimized multi-GPU deployments, reporting ~70x latency cuts and up to 3.6x throughput gains across five applications on B200 and MI355X.

  10. InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbing

    cs.CV 2025-08 conditional novelty 5.0 of 10

    Sparse-frame dubbing with adjacent-chunk keyframe sampling lets a streaming audio-video model produce full-body motion synchronized to new audio while preserving identity and camera motion.

  11. RiverEcho: Real-Time Interactive Digital System for Ancient Yellow River Culture

    cs.MM 2025-06 conditional novelty 4.0 of 10

    The authors built a voice-interactive digital human system for ancient Yellow River culture with a curated 20,000-segment knowledge base and showed that RAG improves answer quality.

Pith tools