{"id":"587fab9c-ace6-4fd3-a068-d80b7c20602a","arxiv_id":"2505.00497","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"KeySync applies a keyframe-interpolated diffusion model with a lower-face mask to generate 512x512 lip-synced video with reduced expression leakage and SAM2-based occlusion handling.","lead":"KeySync is a two-stage lip-sync system that animates 512x512 talking-head videos from a new audio track while masking the lower face to stop the original video's expressions from leaking through. It adds an inference-time occlusion mask from SAM2 and a new metric, LipLeak, to quantify how often silent audio still produces open-mouth frames.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1 may compare 512x512 KeySync outputs to lower-resolution baselines without controlling resolution, so SOTA metric gains could be a resolution artifact.","rationale":"The reader's LipLeak-threshold criticism is valid but not the most load-bearing. Table 1 is the primary evidence for SOTA; without a matched-resolution protocol, even a perfect leakage metric cannot establish the high-resolution claim. The paper never states baselines' output resolutions or whether upscaling was applied before metric computation. VL specifically rewards sharpness, so a native-512 method has an inherent advantage over native-256 methods. The user study also may reflect resolution differences if videos were shown at different sizes. This concern does not invalidate the architectural contributions (two-stage generation, mask design, SAM2 occlusion pipeline), which are coherent and ablated, but it means the headline SOTA claim needs a controlled re-evaluation. A conditional verdict remains appropriate.","tokens_in":19400,"tokens_out":3780,"duration_ms":37772,"concrete_test":"Recompute Table 1 under a controlled resolution: (a) upscale every baseline output to 512x512 (or downsample KeySync to 256x256) before computing CMMD, FVD, VL, LipScore and LipLeak, and (b) rerun the pairwise user study with videos presented at identical display resolutions. If KeySync's margins shrink or invert at matched resolution, the SOTA claim is a resolution artifact rather than a method advantage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"KeySync's central claim is SOTA lip-sync at 512x512, but Table 1 does not report the output resolution of any competing method or state whether outputs were matched before computing metrics. CMMD, FVD and VL are resolution-sensitive: a 512x512 output will generally achieve higher VL and lower FVD/CMMD than the same content at 256x256. If DiffDub, IP_LAP, Diff2Lip, TalkLip or LatentSync were evaluated at their native (typically 256x256 or lower) resolution while KeySync outputs 512x512, the margins in Table 1 and the user-study Elo ratings would be confounded with resolution rather than lip-sync quality. The paper frames 'surpassing the common 256x256 standard' as a contribution, but the comparison protocol never isolates resolution from method quality. This is more load-bearing than the LipLeak threshold issue: if the headline metrics are not computed at matched resolution, the SOTA claim is unsupported regardless of how leakage is measured.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"KeySync proposes a two-stage latent diffusion framework for lip synchronization at 512×512 resolution. The first stage generates sparse keyframes conditioned on HuBERT audio features and an identity frame; the second interpolates between keyframes. Leakage from the input video is addressed by a lower-face mask extending above the nose, and occlusions are handled at inference by excluding SAM2-segmented objects from the mask. The authors introduce LipLeak, a metric measuring the fraction of open-mouth frames when silent audio is fed with non-silent video. Experiments compare KeySync with DiffDub, IP_LAP, Diff2Lip, TalkLip, and LatentSync on reconstruction and cross-sync tasks using CMMD, TOPIQ, VL, FVD, LipScore, LipLeak, and a 1,000-pair user study with Elo ratings. The paper claims state-of-the-art results, particularly in cross-synchronization, and releases code and weights.","tokens_in":19561,"tokens_out":5349,"duration_ms":48928,"significance":"If the claimed results are reproducible at matched resolution, KeySync would be a useful step forward: it is, to my knowledge, one of the first lip-sync systems demonstrated at 512×512, it addresses the under-studied cross-sync setting, and its occlusion-handling trick is simple and applicable to other methods. The paper ships code and weights, performs systematic ablations (architecture, audio encoder, mask, guidance, losses), and grounds its subjective claims in a user study with bootstrap intervals. The LipLeak metric, despite the threshold-sensitivity issue discussed below, is a reasonable first attempt to quantify expression leakage, which is often only discussed qualitatively. However, the central SOTA claim rests on an evaluation protocol that currently does not control for output resolution, does not enforce a speaker-disjoint split, and relies partly on metrics proposed by the same authors; these gaps must be closed before the claim can be accepted.","major_comments":[{"comment":"KeySync is evaluated at 512×512 (Supplement B, Table 7), while the competing methods are, in most cases, native 256×256 systems; Table 1 does not list the output resolution used for each baseline or state whether all outputs were resized to a common resolution before computing CMMD, FVD, VL, or running the user study. All of these metrics are resolution-sensitive (FVD and CMMD compare feature distributions of images of fixed size; the variance of Laplacian directly measures image sharpness). The paper's framing that KeySync 'surpasses the common 256×256 standard' is therefore confounded: the reported margins could reflect a resolution artifact rather than a lip-sync advantage. The authors should either evaluate all methods at a matched resolution (e.g., upscaling baselines to 512×512 or downsampling KeySync to 256×256) or otherwise demonstrate that the conclusions are invariant to resolution; the user-study stimulus resolution should also be reported.","section":"Section 5.1, Table 1"},{"comment":"The 100 test videos are 'randomly selected' from CelebV-Text, CelebV-HQ, and HDTF, the same datasets used for training, and no identity-level split is described. HDTF has only 264 speakers, and random selection at video level makes it very likely that test identities also appear in the training set; if so, identity memorization can inflate image-quality and LipScore numbers. The authors should report whether test identities are disjoint from training identities, and if not, provide results on a held-out identity split. In addition, Table 1 reports single point estimates with no error bars or significance tests; given the two settings and six models, bootstrap confidence intervals on CMMD/FVD/LipScore would be needed to support the claimed ordering.","section":"Section 4.1, test set"},{"comment":"LipLeak is defined by a hand-set MAR threshold of 0.25, chosen 'based on visual inspection.' Figure 10 of the Supplement shows that LipLeak decreases continuously as the threshold increases, so the ranking of models on LipLeak depends on the chosen threshold; the current paper does not show whether the ordering in Table 1 (KeySync 0.16 vs. IP_LAP 0.28 vs. LatentSync 0.33) is stable over a plausible range, e.g., 0.2–0.3. Since LipLeak is used both as a headline contribution and as an ablation criterion (Table 4), the authors should report the metric as a function of threshold for all methods, or provide a principled criterion for the threshold.","section":"Section 4.2 and Supplement C, LipLeak threshold"},{"comment":"LipScore [2] is proposed in KeyFace, which is the same authors' system on which KeySync is built, and the paper uses it as the primary lip-sync quality measure; no independent lip-sync metric (e.g., SyncNet offset and confidence) is reported. Even if LipScore is more effective than SyncNet, its use here means the central claim is partly evaluated with a self-defined metric. The authors should report at least one external, widely used lip-sync measure in Table 1, or demonstrate that LipScore agrees with SyncNet-based scores on a subset of outputs.","section":"Section 4.2, LipScore provenance"}],"minor_comments":[{"comment":"The notation 's ={zti,zm,...,z m| ...' is hard to parse; please define the number of learnable embeddings and clarify the order of frames in the sequence.","section":"Section 3.3, Eq. (3)"},{"comment":"The metric name 'variance of Laplacian (VL)' should be 'variance of the Laplacian' for correctness.","section":"Section 4.2"},{"comment":"The sentence 'it's a a scale-invariant measure' contains a duplicated article and informal contraction; please revise.","section":"Supplement C"},{"comment":"There is a typo in 'would require extesnsive new data collection'; it should be 'extensive'.","section":"Supplement G"},{"comment":"The labels 'Crepeat times' and 'repeat times' appear to be truncated expressions; the repetition count should be stated explicitly.","section":"Figure 2 caption"},{"comment":"The rows 'Ours (nose-level)' and 'Ours' are not clearly defined in the main text; the difference between these two masks should be described explicitly and tied to Figure 7, especially since the text says extending the mask to the eye region is crucial.","section":"Section 5.2, Table 4"},{"comment":"Several references contain stray characters (e.g., 'V ougioukas' in [4]) and inconsistent formatting; the reference list should be cleaned, and the arXiv identifier for KeyFace [2] should be provided if available.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The most serious issue is the resolution mismatch in Table 1; if the authors cannot provide matched-resolution comparisons, the SOTA claim should not stand. I would also encourage the editor to weigh the self-referential metric issue (LipScore from the authors' KeyFace and LipLeak introduced here) when judging the strength of the empirical support. The paper is otherwise well structured and the occlusion-handling idea is neat."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know two things about KeySync. First, the best parts are the masking study and the inference-time SAM2 occlusion handling: they are clean, transferable, and useful. Second, the headline SOTA claim is not yet supported, because Table 1 never controls for output resolution and the test set is drawn from the training datasets without an identity split. That is the short version.\n\nWhat is actually new: the two-stage keyframe diffusion is a direct transplant from the authors' own KeyFace, but the mask comparison is genuinely informative. They show mouth-only masks leak, full lower-face masks hurt quality, and a nose-level mask balances the two. That is a clean empirical result. The occlusion handling is simple, training-free, and they demonstrate it works on several other methods, which is a nice property. LipLeak is a reasonable first attempt at quantifying leakage, though its threshold is hand-set. They also run ablations on audio encoder, guidance, and pixel loss, which is more than many papers in this area do.\n\nThe soft spots are real and addressable. The resolution issue is load-bearing: every competing method generates at 256x256 or lower, KeySync generates at 512x512, and the paper never reports a resolution-matched comparison. VL, FVD, and CMMD are all sensitive to resolution, so the large metric gaps in Table 1 could be partly or largely resolution artifacts. The user study probably also conflates resolution with lip-sync quality. The fix is easy: upscale baselines to 512x512 or downscale KeySync, then rerun the metrics. Second, the 100 test videos are randomly selected from the same datasets used for training, with no speaker-disjoint split. That likely inflates the numbers. Third, LipLeak's MAR threshold of 0.25 is chosen by visual inspection, and the authors' own sensitivity plot shows the metric decreases continuously with threshold; rankings could shift at other thresholds. Fourth, the abstract says code and weights are available, but the supplementary says they will be released upon acceptance. At this stage, the results are not independently reproducible.\n\nThe central idea — a two-stage diffusion model with careful masking can produce high-resolution, low-leakage lip-sync — is plausible and probably useful to the talking-head subfield. I would not desk-reject it. But the evaluation needs heavy revision: release the code, use a speaker-disjoint test split, and redo the comparison at matched resolution. LipLeak should also be validated against human perceptual judgments of leakage.\n\nWho is this for? People working on lip-sync or automated dubbing who want a strong baseline and a useful masking strategy. It deserves a serious referee, but I would want major changes before accepting.","headline":"A solid engineering adaptation of KeyFace to lip-sync, with genuinely useful masking and occlusion-handling results, but the state-of-the-art claim is undercut by a resolution confound and train-test overlap.","tokens_in":20152,"tokens_out":2655,"would_cite":false,"duration_ms":28735,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"KeySync claims state-of-the-art lip synchronization at 512×512 resolution, surpassing the common 256×256 standard and reducing expression leakage from the input video.","keywords":["lip synchronization","audio-driven facial animation","latent diffusion","expression leakage","video inpainting","keyframe interpolation","occlusion handling","mouth aspect ratio"],"falsifier":"Re-run the cross-sync evaluation of Table 1 computing LipLeak at a sweep of mouth-aspect-ratio thresholds (for example 0.15, 0.25, 0.35, 0.45) on the same 100 test videos, and check whether KeySync's rank and margin over LatentSync, Diff2Lip, and IP_LAP survive at every threshold; the paper's Supplement C, Figure 10 already shows LipLeak decreasing continuously with the threshold, so any crossing of the model curves would indicate the reported ordering is an artifact of the 0.25 cutoff. As a control, also compute LipLeak from fully silent input video: if the metric is not near zero in that case, it is measuring natural mouth posture rather than leakage.","tokens_in":19151,"feed_emoji":"👄","tokens_out":9946,"duration_ms":87153,"temperature":0.7,"pith_summary":"The paper claims that lip synchronization can be pushed to 512×512 resolution, double the field's usual 256×256, without the usual trade-offs of temporal flicker and expression leakage. KeySync splits generation into two diffusion stages: sparse keyframes are generated from HuBERT audio embeddings, then interpolated to smooth full-rate video, and a landmark-derived mask covers the lower face up to just above the nose so mouth-movement cues from the input video cannot leak into the output. The authors report better visual quality, lip-sync accuracy, and temporal coherence than five competing methods in cross-sync tests where audio and video come from different recordings, in addition to a new LipLeak metric and a human-preference study that support the leakage reduction. If true, this makes high-resolution automated dubbing practical, including footage where a hand or object covers the mouth, which the method handles at inference time by segmenting the occluder and removing it from the mask.","feed_headline":"Two-stage diffusion reaches 512×512 lip sync with less leakage","feed_subtitle":"KeySync keeps lips matched to new audio and preserves hands or objects covering the mouth.","key_machinery":"The load-bearing mechanism is the two-stage keyframe-and-interpolation schema: a latent diffusion U-Net first generates keyframes spaced twelve frames apart that anchor the phonetic content of the audio, and a second pass reconstructs the frames between each keyframe pair, using a learnable embedding for the missing frames and fusing the HuBERT audio embedding through cross-attention and the diffusion timestep. Two design choices carry the leakage and occlusion claims. The first is a box-style mask, defined from facial landmarks, that covers the lower face and extends slightly above the nose to the bottom of the image, deliberately sized between a mouth-only mask, which leaks mouth cues, and a full lower-face mask, which removes too much context. The second is an inference-time refinement where the facial mask is intersected with the negation of an object segmentation, excluding occluders from the inpainted region. Training uses the EDM latent loss plus a masked L2 loss in RGB space, with only the masked region contributing to the loss.","core_discovery":"KeySync is a two-stage latent diffusion pipeline, built on Stable Video Diffusion and a keyframe-interpolation scheme, that outputs 512×512 lip-synced video. In the first stage a sparse set of keyframes is generated with lip shapes tied to the phonetic content of HuBERT audio embeddings; in the second, the frames between successive keyframes are interpolated, which is what yields temporally smooth motion without extra synchronization losses or past-frame conditioning. Expression leakage is blocked by a box-shaped mask, computed from facial landmarks, that covers the lower face from just above the nose down to the image edge, deliberately between the mouth-only masks that leak and the full lower-face masks that destroy context. An additional L2 loss in RGB space preserves fine mouth detail, and at inference time only, occluding objects are segmented with a video segmentation model and subtracted from the mask so a hand or microphone over the mouth is kept intact. The paper reports that this design beats five recent methods on image and video quality metrics (CMMD, TOPIQ, VL, FVD), lip-sync accuracy (LipScore), the new LipLeak metric, and pairwise human preferences, with the largest margins in the cross-sync setting that matters for dubbing.","pith_inferences":["Testable extension: because the supplement shows LipLeak decreasing continuously as the MAR threshold rises, a threshold sweep on the benchmark videos would settle whether KeySync's leakage advantage over LatentSync and Diff2Lip is a property of the models or an artifact of the hand-picked 0.25 cutoff.","The mask-sizing principle stated in the paper, cover every region that signals mouth movement while keeping everything that preserves identity, generalizes beyond lip-sync to other masked-inpainting editing tasks, such as re-targeting expressions or removing speech-related motion from archival footage.","Since the occlusion fix is a pure inference-time mask operation, it is a portable module: the paper demonstrates it on DiffDub and Diff2Lip out of the box, so it could plausibly become a standard pre-processing step for any free-form-mask lip-sync model.","The silent-audio protocol assumes the ideal output for silent speech is a closed mouth; a stress test with speakers who naturally keep their mouths slightly open when quiet would reveal whether LipLeak partly confounds natural mouth posture with leakage."],"forward_implications":["Lip-synced output can be produced at 512×512, doubling the linear resolution of the field's de facto 256×256 standard, and human raters prefer it in both reconstruction and cross-sync settings.","In cross-sync tests, where the driving audio comes from a different recording than the face, KeySync's lip-sync accuracy stays nearly constant while most baselines drop sharply, which is exactly the regime automated dubbing operates in.","Expression leakage can be measured directly: feeding silent audio with non-silent video and counting open-mouth frames via the mouth aspect ratio yields a metric that ranks models and exposes leakage that lip-sync scores can misread.","Occlusions such as hands, microphones, or motion blur over the mouth can be handled without retraining by segmenting the occluder and removing it from the model's mask, reducing reconstruction error around the occluder.","Temporal consistency follows from the keyframe-then-interpolate decomposition itself, so the approach needs no sequence discriminator, temporal perceptual loss, or conditioning on past frames to stay smooth."],"supporting_citations":[{"why":"Supplies the two-stage keyframe-and-interpolation framework that KeySync adapts from facial animation to lip-sync.","marker":"[2]"},{"why":"Stable Video Diffusion provides the pre-trained latent diffusion U-Net and VAE that both generation stages are initialized from.","marker":"[4]"},{"why":"HuBERT encodes the driving audio, with its embeddings fused into the U-Net through cross-attention and the timestep embedding.","marker":"[22]"},{"why":"The EDM formulation defines the latent diffusion loss and noise-dependent weighting used to train both stages.","marker":"[27]"},{"why":"The facial landmark detector is used to compute the lower-face mask and the mouth aspect ratio inside LipLeak.","marker":"[5]"},{"why":"The video segmentation model (SAM 2) generates occlusion masks that are subtracted from the inpainting mask at inference.","marker":"[39]"},{"why":"Wav2Lip is the foundational lip-sync baseline whose mask convention and lip-sync expert the paper positions itself against.","marker":"[36]"},{"why":"LatentSync is the strongest diffusion-based baseline, providing the main comparison for quality, leakage, and human preference.","marker":"[29]"}],"fun_headline_variants":["KeySync: Leakage-free lip sync that preserves occlusions","Two-stage diffusion for occlusion-ready lip sync at 512×512","KeySync: Less leakage, occlusion handling, state-of-the-art sync","New metric and mask stop lip sync expression leakage","KeySync: High-res lip sync without expression leakage"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole leakage story rests on LipLeak's open-mouth threshold: a mouth aspect ratio of 0.25, chosen by visual inspection, is treated as the dividing line between a mouth that leaked from the input video and a mouth that did not, and the method rankings the paper reports are sensitive to that number since the supplement shows LipLeak falling continuously as the threshold rises.","fun_headline_variants_meta":{"raw":{"variants":["KeySync: Leakage-free lip sync that preserves occlusions","Two-stage diffusion for occlusion-ready lip sync at 512×512","KeySync: Less leakage, occlusion handling, state-of-the-art sync","New metric and mask stop lip sync expression leakage","KeySync: High-res lip sync without expression leakage"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000216,"raw_usage":{"total_tokens":1448,"prompt_tokens":979,"completion_tokens":469,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":595,"completion_tokens_details":{"reasoning_tokens":384}},"tokens_in":595,"tokens_out":469,"duration_ms":4965,"temperature":1.0,"reasoning_tokens":384,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:40:07.126007+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the cross-sync evaluation of Table 1 computing LipLeak at a sweep of mouth-aspect-ratio thresholds (for example 0.15, 0.25, 0.35, 0.45) on the same 100 test videos, and check whether KeySync's rank and margin over LatentSync, Diff2Lip, and IP_LAP survive at every threshold; the paper's Supplement C, Figure 10 already shows LipLeak decreasing continuously with the threshold, so any crossing of the model curves would indicate the reported ordering is an artifact of the 0.25 cutoff. As a control, also compute LipLeak from fully silent input video: if the metric is not near zero in that case, it is measuring natural mouth posture rather than leakage.","supporting_citations":[{"cited_title":"Keyface: Expressive audio-driven facial animation for long sequences via keyframe interpolation, 2025","cited_arxiv_id":null,"evidence_quote":"Supplies the two-stage keyframe-and-interpolation framework that KeySync adapts from facial animation to lip-sync."},{"cited_title":"Stable video diffusion: Scaling latent video diffusion models to large datasets, 2023","cited_arxiv_id":null,"evidence_quote":"Stable Video Diffusion provides the pre-trained latent diffusion U-Net and VAE that both generation stages are initialized from."},{"cited_title":"Hubert: Self-supervised speech representation learning by masked prediction of hidden units","cited_arxiv_id":null,"evidence_quote":"HuBERT encodes the driving audio, with its embeddings fused into the U-Net through cross-attention and the timestep embedding."},{"cited_title":"Elucidating the design space of diffusion-based generative models","cited_arxiv_id":null,"evidence_quote":"The EDM formulation defines the latent diffusion loss and noise-dependent weighting used to train both stages."},{"cited_title":"How far are we from solving the 2d & 3d face alignment problem? (and a dataset of 230, 000 3d facial landmarks)","cited_arxiv_id":null,"evidence_quote":"The facial landmark detector is used to compute the lower-face mask and the mouth aspect ratio inside LipLeak."},{"cited_title":"A lip sync expert is all you need for speech to lip generation in the wild","cited_arxiv_id":null,"evidence_quote":"Wav2Lip is the foundational lip-sync baseline whose mask convention and lip-sync expert the paper positions itself against."}],"review_version":1}