Pith. sign in

REVIEW 5 major objections 6 minor 38 references

Video Editing for Audio-Visual Dubbing

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Dubbing as video editing, not generation, preserves occlusions and identity while lips follow new audio.

desk verdict EdiDub is a solid dubbing system with a genuinely new conditioning scheme, but the paper's central 'editing, not inpainting' claim is not actually measured. read the letter →

arxiv 2505.23406 v1 pith:4GP6BYDI submitted 2025-05-29 cs.CV cs.AIcs.LG

Binyamin Manela , Sharon Gannot , Ethan Fetyaya This is my paper
classification cs.CVcs.AIcs.LG
keywords visualdubbingcontent-awareeditinglipsynchronizationdiffusionmodelDDIMinversionidentitypreservationHuBERTaudiofeaturesvideo
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that visual dubbing should be reformulated as a content-aware editing task rather than a generation or inpainting task. The authors introduce EdiDub, a diffusion-based framework that conditions on the full original video and uses DDIM inversion to initialize the edit, so that only the articulatory motion changes while occlusions, lighting, and identity stay fixed. They report that this approach outperforms Wav2Lip, LatentSync, and DiffDub on lip-sync accuracy and identity preservation, with the largest gains on a benchmark where hands occlude the lips. Human raters also preferred EdiDub for synchronization and naturalness. If correct, this means the key to realistic dubbing is not to redraw the mouth from scratch but to edit the existing footage as little as possible.

What carries the argument

Three components carry the argument. The first is an optimized visual reference: during training, each frame is paired with a same-identity frame at least five frames away whose facial landmarks, excluding the lips, are closest; this teaches the model to copy spatial details while ignoring the reference mouth, and the same mechanism transfers to inference where the reference is the original frame. The second is DDIM inversion, which converts the original clip conditioned on its own speech into an initial noise latent, so the edit starts from a faithful reconstruction. The third is a quantized HuBERT speech representation with a single-speaker codebook, injected through adaptive instance normalization, which provides frame-aligned phonetic conditioning without cross-attention layers.

What would settle it

Take a trained model and condition it on a reference frame whose lips are visibly speaking a different utterance than the target audio; if the output reproduces the reference mouth rather than tracking the new audio, the 'learn to ignore the reference lips' assumption is false. Alternatively, dub a video with its own original audio as the target: if the output is nearly identical to the input, the model may have learned to copy rather than to edit.

Watch

Extended reading notes

Core claim

The central claim is that conditioning a diffusion model on the original unmasked video frames, together with a frame-accurate speech representation, lets the model alter only the lip motion while faithfully preserving the rest of the visual scene. EdiDub achieves this with a training-time reference-frame selection scheme that prevents the degenerate copy solution, and with DDIM inversion at inference so that generation starts from a noise code that already encodes the original clip's structure. The paper demonstrates on LRS2, LRS3, and two VoxCeleb2-derived benchmarks that this editing formulation achieves lower lip-sync error, higher sync confidence, and better identity preservation than generation- and inpainting-based baselines, especially when the mouth area is partially occluded.

Load-bearing premise

The central assumption is that training with a same-identity reference frame whose lips do not match the audio teaches the model to ignore that reference's mouth motion while still using its spatial details, and that this behavior transfers to inference, where the reference is the original video frame itself.

Editorial extensions

If this is right

  • Dubbing can preserve scene elements such as hands covering the mouth, motion blur, and lighting variations that inpainting methods erase.
  • Identity preservation and lip synchronization need not be strongly opposing objectives when the model is anchored to the original video through conditioning and inversion.
  • A single-speaker audio codebook combined with the known audio-to-video frame alignment is sufficient to drive accurate articulation, removing the need for costly cross-attention.
  • On the LRS3 benchmark, the method's audio-visual synchronization metrics are on par with real recordings, suggesting the task is nearing a perceptual ceiling for that dataset.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reference-frame selection principle implies that any conditioning signal carrying the same identity but different mouth motion could substitute for the original frame, so the method might extend to dubbing videos that have no clean reference frame of the speaker.
  • The same content-aware editing formulation could transfer to other face-region edits, such as changing expression or apparent age while leaving the rest of the frame untouched; the paper does not explore this.
  • The ablation shows a clear faithfulness-versus-articulation trade-off when inversion or reference selection is removed, suggesting the guidance scale or inversion strength could be a user-facing dial for that trade-off, which the paper leaves implicit.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes EdiDub, a two-stage diffusion system for visual dubbing. The first stage (LSD) edits a 64x64, 25-frame clip by masked diffusion conditioned on the full original video plus quantized HuBERT speech units, using DDIM inversion and MultiDiffusion for long videos; the second stage (SRD) restores 224x224 detail. The claimed contribution is reformulating dubbing as content-aware editing, rather than generation or inpainting, in order to preserve occlusions, lighting, and identity while changing only articulatory lip motion. The evaluation compares with Wav2Lip, DiffDub, and LatentSync on LRS2, LRS3, and two new VoxCeleb2 benchmarks (front-facing and occluded), reporting lip-sync metrics (LSE-D/C), two identity metrics (ID-P/ID-TC), MOS ratings with standard errors, and ablations of inversion, reference selection, and HuBERT clustering.

Significance. The task is practically important and the method is broadly sound in its engineering: paired Wilcoxon tests are reported, the MOS protocol includes quality-control screening, the occluded-lip benchmark is a useful new resource, and the architectural hyperparameters are concrete enough to re-implement. If the 'content-aware editing' claim can be supported by direct measurements, the paper would be a meaningful advance over inpainting-style dubbing, particularly for occluded and in-the-wild content. As submitted, however, the evidence does not yet close the gap between the central claim and the reported metrics, so the significance is conditional on the additional measurements requested below.

major comments (5)
  1. [§4.1, Table 3, Fig. 1] The central claim that EdiDub performs faithful content-aware editing rather than copying (Abstract, §1) is not directly supported by any reported metric. ID-P is a whole-face cosine distance between embeddings of original and generated frames; a no-op that returns the original video achieves ID-P near zero by construction, and MOS-Nat also rewards an unmodified original. The unique selling point on Vox2-Occluded is preservation of hand/object occlusions over the lips, but neither ID-P nor MOS-Nat measures whether the occlusion is retained; the qualitative claim in Fig. 1 is not quantified. The ablations in Table 3 even show that removing DDIM inversion or optimized reference selection improves LSE-D/LSE-C while degrading ID-P, which is consistent with the hypothesis that higher ID-P reflects copying more of the original rather than content-aware editing. A no-op baseline (dubbing with the original audio) and a region-wise fidelity metric restricted to non-articulatory areas are needed to separate context preservation from copy behavior. Without these, the abstract's contrast with inpainting is not established.
  2. [§3.2] The SRD input at inference is not fully specified. The text states that at each denoising step SRD receives a concatenated tensor containing 'the masked high-resolution ground-truth frames' yGT_0 together with the upsampled low-resolution reference, and that only the low-resolution component is replaced by the LSD output at inference. Since no ground truth exists at inference, this description is either a typo or implies that the original high-resolution frames (with the edited region masked) are used as context. If the latter is intended, it should be stated explicitly; as written, the final 224x224 stage is not reproducible from the description and appears ill-posed.
  3. [§3.1, Eq. (2)] The DDIM inversion update in Eq. (2) is not the standard DDIM inversion step. Standard inversion is an approximate reverse of the forward process and involves a ratio of alpha terms and an additive noise term involving epsilon_theta; Eq. (2) as written resembles a deterministic denoising update with the time index shifted rather than an inversion. Because DDIM inversion is a core component of the claimed editing behavior and is used in all reported results, the formula must be corrected or its notation justified, and the effect of using this approximate inversion should be discussed.
  4. [§3.1, 'Accurate visual reference'] The training reference strategy rests on an unmeasured transfer assumption. During training the model is conditioned on a same-identity reference frame at least five frames away so that it must ignore reference lip position, while at inference it is conditioned on the original video frame itself, whose lips are misaligned with the target audio. If the model over-relies on the reference lips, it copies the original misaligned mouth and lip-sync metrics improve spuriously; if it suppresses the conditioning too much, occlusions and context are lost. The paper does not test this transfer directly, and the ablations in Table 3 only vary the reference selection strategy, not the conditioning content at inference. A diagnostic that measures how strongly the model's output lip region correlates with the input/reference lip region, or a comparison against a model trained with the inference-time conditioning protocol, would make the central mechanism credible.
  5. [§3, first paragraph; Abstract] The first sentence of §3 says 'Both stages are trained as masked video inpainting problems.' This directly undercuts the Abstract's claim that the method 'reformulates visual dubbing as a content-aware editing task' and the Introduction's contrast with inpainting. The distinction is not merely rhetorical: if the training objective and mask-based diffusion are those of inpainting, the paper should state specifically what makes inference 'editing' beyond conditioning on the original video, and the evaluation should measure the additional property it claims. Please either revise the framing or add the missing measurement.
minor comments (6)
  1. [§4.4] The first sentence says 'Our method receives the highest MOS-Sync and MOS-Sync scores across all datasets'; the second score should be MOS-Nat.
  2. [§4.1 vs Appendix C.2] Section 4.1 says Vox2-Front clips have facial orientations within ±25 degrees from frontal, but Appendix C.2 and Algorithm 1 reject videos with a pose score exceeding 20 degrees; these thresholds should be reconciled.
  3. [Appendix A.1] In the 'Real Videos' rows, ID-P is 0.000 by construction because the original video is compared with itself, so it is not a meaningful anchor for the claim that a method is 'closest to real videos'; this caveat should be stated where the comparison is made in Section 4.3.
  4. [§4.2] The long-video stitching description ('Each section is generated sequentially, with the leading 12 frames from the end of the previous section with no mask (mask of zeros)') is unclear; please specify which frames are conditioned and which are generated when sections overlap.
  5. [Appendix B, Table 6] The SRD entries 'Masked MSE (MMSE) | MSE' and 'Mask lower-face area | None' conflict with the statement in §3.2 that SRD is trained with the same masked-reconstruction loss as LSD; the table and text should be aligned.
  6. [§4.1 and Appendix B] The paper says the code 'will be made available in the supplementary material' (Section 4.1) and also that it 'will be added as soon as possible' to an anonymous repository (Appendix B); the exact availability status should be stated in the final version.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: EdiDub is an empirical learned system; its identity-preservation metric can reward copying, but that is an evaluation limitation, not a by-construction derivation.

full rationale

The paper does not derive a law or fit a parameter and then rename it as a prediction; it trains a conditional diffusion model and evaluates it on external benchmarks. The DDIM-inversion step (Eq. 2) is a standard invert-then-denoise editing heuristic, and the statement that it preserves non-lip content is an empirical assumption, not a consequence of the equation by construction. The reference-frame selection scheme is a training strategy; at inference the model sees the original video, so low ID-P could in principle be achieved by copying, but the paper also reports external LSE-D/C and human MOS, and the ablations (Table 3) show that removing the identity-anchoring components degrades ID-P while improving LSE, so the headline results are not forced solely by the conditioning input. There are no load-bearing self-citations, no imported uniqueness theorem, and no fitted constant presented as a prediction; the references cited for HuBERT clustering, DDIM inversion, and MultiDiffusion are external. The main flagged concern is internal consistency rather than circularity: Section 3 states 'Both stages are trained as masked video inpainting problems,' which undercuts the Abstract's contrast with inpainting, and the claimed preservation of occlusions on Vox2-Occluded is not directly quantified. These are evaluation and novelty limitations, not circular derivation steps.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The system is an empirical machine learning method, so its central claim rests on the training data distribution, the reference-frame training scheme, and the DDIM inversion behavior. No new physical or mathematical entities are introduced. The main free parameters are hand-chosen hyperparameters that affect the final performance.

free parameters (6)
  • HuBERT codebook size = k=200
    Set by hand in Sec. 3.1 and App. B.1; controls phonetic resolution and the dimensionality of audio conditioning.
  • Classifier-free guidance scale = s=5
    Chosen in Sec. 3.1 and Table 5; amplifies audio conditioning during sampling and affects the identity-sync trade-off.
  • DDIM steps = T=50
    Used for both inversion and sampling (Sec. 3.1, Table 5); number of diffusion steps, chosen for quality and speed.
  • MultiDiffusion window and step = 24-frame window, 12-frame step
    Window and overlap for long-video stitching (Sec. 4.2, Table 5); chosen without systematic search.
  • Reference exclusion window = ±5 frames
    Minimum temporal distance between a frame and its training reference (Sec. 3.1); chosen to prevent trivial matches and affects the train/inference gap.
  • Mask region = below the eyes
    Defines the area to be regenerated (Sec. 3.1); chosen by hand instead of learned, and the central claim depends on this region being the right one.
assumptions (4)
  • domain assumption HuBERT features quantized with a single-speaker LJ-Speech codebook remove speaker identity and preserve phonetic content.
    Used in Sec. 3.1 to justify the audio representation; relies on [19] but the paper does not verify this for its own setting.
  • ad hoc to paper Training with reference frames from the same identity teaches the model to ignore reference lip movement while using spatial context, and this transfers to inference with original-video conditioning.
    The central assumption of the training scheme (Sec. 3.1); not proven, only ablated indirectly through ID-P.
  • standard math DDIM inversion reconstructs the original video's structure faithfully when conditioning on original speech, so editing with new speech preserves context.
    Assumed from [9] in Sec. 3.1; standard behavior of DDIM, but the paper does not analyze failure cases.
  • domain assumption The 25-frame and 5-frame training windows, and the LRS3 training set, are sufficient to generalize to the four evaluation benchmarks.
    Training details in Sec. 4.2 and App. B; no analysis of domain shift from LRS3 to VoxCeleb2.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Video Editing for Audio-Visual Dubbing." pith.science (2026). https://pith.science/paper/4GP6BYDI

@misc{pith2026250523406,
  author       = {Pith},
  title        = {Pith review of: Video Editing for Audio-Visual Dubbing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4GP6BYDI}},
  note         = {Machine review of arXiv:2505.23406}
}
read the original abstract

Visual dubbing, the synchronization of facial movements with new speech, is crucial for making content accessible across different languages, enabling broader global reach. However, current methods face significant limitations. Existing approaches often generate talking faces, hindering seamless integration into original scenes, or employ inpainting techniques that discard vital visual information like partial occlusions and lighting variations. This work introduces EdiDub, a novel framework that reformulates visual dubbing as a content-aware editing task. EdiDub preserves the original video context by utilizing a specialized conditioning scheme to ensure faithful and accurate modifications rather than mere copying. On multiple benchmarks, including a challenging occluded-lip dataset, EdiDub significantly improves identity preservation and synchronization. Human evaluations further confirm its superiority, achieving higher synchronization and visual naturalness scores compared to the leading methods. These results demonstrate that our content-aware editing approach outperforms traditional generation or inpainting, particularly in maintaining complex visual elements while ensuring accurate lip synchronization.

Figures

Figures reproduced from arXiv: 2505.23406 by the authors.

Figure 1
Figure 1. Qualitative comparison across time. Each row presents 7 consecutive frames from a different method, with the original video at the top and our model at the bottom. Existing methods create severe artifacts on these examples where the mouth is occluded. on producing realistic and convincing videos but often failing to blend seamlessly within an existing visual scene. These methods typically generate impressive isolate… view at source ↗
Figure 2
Figure 2. Overview of our dubbing system EdiDub. (a) Model architectures for both stages of our pipeline. (b) End-to-end inference process for lip-synced facial video generation. 3.1 Lip-Sync Diffusion (LSD) Masked DDPM objective. Let x0, m∈R 25×64×64×3 be a 25-frame RGB clip and the binary mask for the facial area below the speaker’s eyes, respectively. During the forward diffusion process, we add Gaussian noise only inside … view at source ↗
Figure 3
Figure 3. MOS evaluation interface presented to participants. Each task consists of six randomized [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Example frame from a corrupted video used for attention screening. The mouth region [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

38 extracted references · 18 canonical work pages

  1. [1]

    What is the McGurk effect?Frontiers in psychology, 2014

    Kaisa Tiippana. What is the McGurk effect?Frontiers in psychology, 2014

    work page 2014

  2. [2]

    Denoising diffusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020

    2020

  3. [3]

    FlowVQTalker: High-quality emotional talking face generation through normalizing flow and quantization

    Shuai Tan, Bin Ji, and Ye Pan. FlowVQTalker: High-quality emotional talking face generation through normalizing flow and quantization. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26317–26327, 2024

    work page 2024

  4. [4]

    Diffused heads: Diffusion models beat GANs on talking-face generation

    Michał Stypułkowski, Konstantinos V ougioukas, Sen He, Maciej Zi˛ eba, Stavros Petridis, and Maja Pantic. Diffused heads: Diffusion models beat GANs on talking-face generation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 5091–5100, 2024

    work page 2024

  5. [5]

    EmoTalker: Emotionally editable talking face generation via diffusion model

    Bingyuan Zhang, Xulong Zhang, Ning Cheng, Jun Yu, Jing Xiao, and Jianzong Wang. EmoTalker: Emotionally editable talking face generation via diffusion model. InIEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8276–8280, 2024. 9

    work page 2024

  6. [6]

    LatentSync: Audio conditioned latent diffusion models for lip sync.arXiv preprint arXiv:2412.09262, 2024

    Chunyu Li, Chao Zhang, Weikai Xu, Jinghui Xie, Weiguo Feng, Bingyue Peng, and Weiwei Xing. LatentSync: Audio conditioned latent diffusion models for lip sync.arXiv preprint arXiv:2412.09262, 2024

    arXiv 2024

  7. [7]

    A lip sync expert is all you need for speech to lip generation in the wild

    KR Prajwal, Rudrabha Mukhopadhyay, Vinay P Namboodiri, and CV Jawahar. A lip sync expert is all you need for speech to lip generation in the wild. InProceedings of the 28th ACM international conference on multimedia, pages 484–492, 2020

    2020

  8. [8]

    DiffDub: Person-generic visual dubbing using inpainting renderer with diffusion auto-encoder

    Tao Liu, Chenpeng Du, Shuai Fan, Feilong Chen, and Kai Yu. DiffDub: Person-generic visual dubbing using inpainting renderer with diffusion auto-encoder. InIEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3630–3634, 2024

    work page 2024

Show all 38 references
  1. [9]

    Denoising diffusion implicit models.arXiv preprint arXiv:2010.02502, 2020

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models.arXiv preprint arXiv:2010.02502, 2020

    2010 arXiv

  2. [10]

    Hubert: Self-supervised speech representation learning by masked prediction of hidden units.IEEE/ACM transactions on audio, speech, and language processing, 29:3451–3460, 2021

    Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units.IEEE/ACM transactions on audio, speech, and language processing, 29:3...

    2021

  3. [11]

    Video diffusion models.Advances in Neural Information Processing Systems, 35:8633–8646, 2022

    Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models.Advances in Neural Information Processing Systems, 35:8633–8646, 2022

    2022

  4. [12]

    Cascaded diffusion models for high fidelity image generation.Journal of Machine Learning Research, 23(47):1–33, 2022

    Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation.Journal of Machine Learning Research, 23(47):1–33, 2022

    2022

  5. [13]

    V oxceleb2: Deep speaker recognition

    Joon Son Chung, Arsha Nagrani, and Andrew Zisserman. V oxceleb2: Deep speaker recognition. arXiv preprint arXiv:1806.05622, 2018

    2018 arXiv

  6. [14]

    Stylesync: High-fidelity generalized and personalized lip sync in style-based generator

    Jiazhi Guan, Zhanwang Zhang, Hang Zhou, Tianshu Hu, Kaisiyuan Wang, Dongliang He, Haocheng Feng, Jingtuo Liu, Errui Ding, Ziwei Liu, et al. Stylesync: High-fidelity generalized and personalized lip sync in style-based generator. InProceedings of the IEEE/CVF Conference on Comp...

    2023

  7. [15]

    Analyzing and improving the image quality of stylegan

    Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8110–8119, 2020

    2020

  8. [16]

    Diff2lip: Audio conditioned diffusion models for lip-synchronization

    Soumik Mukhopadhyay, Saksham Suri, Ravi Teja Gadde, and Abhinav Shrivastava. Diff2lip: Audio conditioned diffusion models for lip-synchronization. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5292–5302, 2024

    2024

  9. [17]

    High- resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022

    2022

  10. [18]

    The LJ speech dataset

    Keith Ito and Linda Johnson. The LJ speech dataset. https://keithito.com/ LJ-Speech-Dataset/, 2017

    2017

  11. [19]

    Revise: Self-supervised speech resynthesis with visual input for universal and generalized speech regeneration

    Wei-Ning Hsu, Tal Remez, Bowen Shi, Jacob Donley, and Yossi Adi. Revise: Self-supervised speech resynthesis with visual input for universal and generalized speech regeneration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18795–18...

    2023

  12. [20]

    Arbitrary style transfer in real-time with adaptive instance normalization

    Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. InProceedings of the IEEE International Conference on Computer Vision (ICCV), pages 1501–1510, 2017

    2017

  13. [21]

    Classifier-free diffusion guidance.arXiv preprint arXiv:2207.12598, 2022

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance.arXiv preprint arXiv:2207.12598, 2022. 10

    2022 arXiv

  14. [22]

    Multidiffusion: Fusing diffusion paths for controlled image generation

    Omer Bar-Tal, Lior Yariv, Yaron Lipman, and Tali Dekel. Multidiffusion: Fusing diffusion paths for controlled image generation. 2023

    2023

  15. [23]

    Lip reading sentences in the wild

    Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman. Lip reading sentences in the wild. InProceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pages 6447–6456, 2017

    2017

  16. [24]

    LRS3-TED: A large-scale dataset for visual speech recognition.arXiv preprint arXiv:1809.00496, 2018

    Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman. LRS3-TED: A large-scale dataset for visual speech recognition.arXiv preprint arXiv:1809.00496, 2018

    2018 arXiv

  17. [25]

    Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset

    Zhimeng Zhang, Lincheng Li, Yu Ding, and Changjie Fan. Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3661–3670, 2021

    2021

  18. [26]

    ModeFormer: Modality-preserving embed- ding for audio-video synchronization using transformers

    Akash Gupta, Rohun Tripathi, and Wondong Jang. ModeFormer: Modality-preserving embed- ding for audio-video synchronization using transformers. InIEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5, 2023

    2023

  19. [27]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational Conference on Machine Learning (ICM...

    2021

  20. [28]

    J. S. Chung and A. Zisserman. Out of time: Automated lip sync in the wild. InWorkshop on Multi-view Lip-reading, ACCV, 2016

    2016

  21. [29]

    facenet-pytorch.https://github.com/timesler/facenet-pytorch, 2022

    Tim Esler. facenet-pytorch.https://github.com/timesler/facenet-pytorch, 2022

    2022

  22. [30]

    Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  23. [31]

    Wilcoxon signed-rank test.Encyclopedia of biostatistics, 8, 2005

    Robert F Woolson. Wilcoxon signed-rank test.Encyclopedia of biostatistics, 8, 2005

    2005

  24. [32]

    Diffusion models beat gans on image synthesis

    Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780–8794, 2021

    2021

  25. [33]

    guided-diffusion.https://github.com/openai/guided-diffusion, 2021

    OpenAI. guided-diffusion.https://github.com/openai/guided-diffusion, 2021

    2021

  26. [34]

    Improved denoising diffusion probabilistic models

    Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. InInternational conference on machine learning, pages 8162–8171. PMLR, 2021

    2021

  27. [35]

    ns" represents non-significant results (p > 0.05),

    google-ai edge. mediapipe.https://github.com/google-ai-edge/mediapipe, 2025. 11 A Extended Results In this section, we provide a deeper look into the results of our experiments, including statistical tests, our MOS web page, and our MOS results analysis process. All the videos...

    2025

  28. [36]

    We used a constant crop to ensure the corrupted video would not inadvertently synchronize with the audio

    Replacing the mouth region with a constant crop from a different frame, updating this crop once every 25 frames. We used a constant crop to ensure the corrupted video would not inadvertently synchronize with the audio

  29. [37]

    Rotating the inserted crop by 180 degrees

  30. [38]

    facebook/hubert-large-ll60k

    Adding unique Gaussian noise (mean 0, standard deviation 20) to the pasted region in every frame. These distortions were deliberately applied only after the first 15 frames and terminated 5 frames before the end, ensuring that participants needed to watch a significant portion...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.