{"id":"447b2f96-2aca-49fb-9a9f-1d0b216a1842","arxiv_id":"2507.05092","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"MoDiT, a diffusion transformer conditioned on 3DMM coefficients and Wav2Lip references, produces talking-head videos with improved same-identity lip sync and more natural blinks in its reported benchmarks.","lead":"MoDiT is a new audio-driven talking head model that combines a 3D face model with a diffusion transformer to create lip-synced video with fewer temporal jumps and more natural blinking. Its benchmark results are competitive with recent systems on HDTF, VFHQ, and VoxCeleb-HQ, though the gains are small and some evaluation choices are circular.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1's Wav2Lip baseline outscores MoDiT on both LSE metrics, and MoDiT's renderer embeds Wav2Lip; the claimed same-identity lip-sync superiority is therefore internally contradicted and possibly inherited.","rationale":"I read MoDiT as a conditional diffusion-transformer model over 3DMM expression coefficients, with a hierarchical bias schedule, revised temporal attention, a blink module, and a flow-based renderer that injects Wav2Lip output. For the central claim to hold, the favorable lip-sync numbers in Table 1 must be attributable to MoDiT's own motion prediction, not to the bundled Wav2Lip reference. That condition is the least secure part of the paper: the evaluation metric is the same family of SyncNet-style scores that Wav2Lip was trained to maximize, and the only relevant ablation (Table 4) removes an entire UNet rather than cleanly disabling the Wav2Lip input. The table also undermines the wording of the claim, since Wav2Lip itself has better LSE-C/LSE-D. I do not see this as a reason to reject the architecture; the method may well work, but the quantitative support is currently ambiguous. The blink and temporal-consistency evidence are weaker still, but the lip-sync claim is the load-bearing assertion in the abstract, Section 4.1, and the reader's strongest_claim. A conditional acceptance with a mandatory Wav2Lip-attribution experiment is the appropriate disposition.","tokens_in":11787,"tokens_out":7742,"duration_ms":86481,"concrete_test":"Re-run the HDTF Same-Identity evaluation of the full MoDiT pipeline with the Wav2Lip branch disabled (feed the original source frame, or a zeroed/blank lip crop, into the Reference UNet in place of the Wav2Lip reference) while keeping the 3DMM diffusion transformer, mapping net, and all losses fixed; recompute LSE-C and LSE-D. If LSE-C stays above roughly 7.3 and LSE-D below roughly 7.8, the Table 1 lip-sync numbers are not primarily inherited from Wav2Lip; if the scores fall to the 'w/o Reference Unet' level of Table 4 (6.898 / 8.273) or below, the Section 4.1 'best lip-sync' claim should be re-scoped or withdrawn. This check requires either the authors' code or a faithful re-implementation, since no code is provided.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.4.5 makes Wav2Lip an input to the Renderer's Reference UNet, so Wav2Lip's output is part of every generated frame. Wav2Lip is trained with a SyncNet-based lip-sync loss, which is the same family of measure as the reported LSE-C/LSE-D. Table 1 (HDTF Same-Identity) lists Wav2Lip at LSE-C 9.652 / LSE-D 5.656 and MoDiT at 7.428 / 7.645; VFHQ Same-Identity shows the same pattern (7.982 / 6.964 vs 6.205 / 7.437). The Section 4.1 statement that MoDiT 'demonstrates the best lip-sync performance across the Same-Identity on both datasets' is therefore not supported by the table as printed unless Wav2Lip is silently excluded from the comparison. More fundamentally, because the comparison methods (Hallo, DreamTalk, AniPortrait, SadTalker, MakeItTalk) do not contain Wav2Lip, the reported margin over them may be an artifact of injecting a lip-sync-optimized reference rather than a property of MoDiT's 3DMM diffusion transformer. The ablation in Table 4 removes the whole Reference UNet rather than toggling the Wav2Lip reference alone, so it cannot separate the two effects.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes MoDiT, a talking-head generation method that combines 3DMM expression coefficients with a diffusion transformer. It introduces a hierarchical denoising strategy with biased self/cross-attention, a revised temporal attention with a learned latent modifier, a blink module with a learnable blink sequence, and a renderer that uses Wav2Lip's output as a reference for a U-Net that predicts optical flow. Training uses an expression diffusion network with noise, velocity, lip-reading, and landmark losses. Evaluations on HDTF, VFHQ, and VoxCeleb-HQ report LSE-C/LSE-D, FID, LMD, AUE, and inference time, with claims of best same-identity lip-sync and SOTA-comparable consistency.","tokens_in":12177,"tokens_out":5697,"duration_ms":55961,"significance":"If the claims were supported, MoDiT would make a meaningful contribution: it injects explicit 3DMM spatial constraints into a diffusion-transformer backbone, introduces a curriculum-like bias strategy for lip focus, and proposes a learned temporal attention modification and blink modeling. The paper also provides an out-of-distribution evaluation and several external metrics (FID, LMD, AUE) that are not tied to Wav2Lip. However, the central lip-sync claim is undermined by the inclusion of Wav2Lip in Table 1 with better LSE scores, by the use of Wav2Lip as an input to the renderer, and by conflicting ablation values between Table 3 and Table 4. The contribution is therefore plausible but not demonstrated as claimed.","major_comments":[{"comment":"The claim that MoDiT demonstrates the best lip-sync performance across Same-Identity on both datasets is contradicted by the table as printed: for HDTF Same-Identity, Wav2Lip achieves LSE-C 9.652 / LSE-D 5.656 versus MoDiT's 7.428 / 7.645, and for VFHQ Same-Identity Wav2Lip achieves 7.982 / 6.964 versus 6.205 / 7.437. Since Section 3.4.5 makes Wav2Lip's output an input to the Reference UNet of the renderer, the reported LSE margins over the other methods may reflect an injected lip-sync-optimized reference rather than the proposed 3DMM diffusion transformer. Please remove Wav2Lip from the comparison or provide a direct same-protocol comparison showing MoDiT outperforms it without relying on Wav2Lip as a reference.","section":"Table 1 / Section 4.1"},{"comment":"This ablation removes the entire Reference UNet, but Section 3.4.5 says Wav2Lip's output is the input to the Reference UNet. The experiment therefore cannot separate the contribution of the proposed flow/3DMM pipeline from the contribution of the Wav2Lip reference itself. In addition, the same ablation row (w/o 3DMM source β0) reports LSE-C 5.365 / LSE-D 9.431 in Table 3 but LSE-C 1.478 / LSE-D 13.596 in Table 4; both cannot be correct for the same configuration. These discrepancies need to be reconciled before the renderer ablation can support the claim.","section":"Section 4.2.2 / Table 4"},{"comment":"The sentence 'Removing the 3DMM coefficient constraints β0 significantly reduced spatial coherence and affected lip synchronization (low LSE-D, high LSE-C)' reverses the direction of the metrics: for the w/o 3DMM row, LSE-C drops from 7.428 to 5.365 and LSE-D rises from 7.645 to 9.431, i.e., the model is less confident and more distant, so the parenthetical should read 'low LSE-C, high LSE-D'. As written, the text reports the opposite of the tabulated result.","section":"Section 4.2.1 / Table 3"},{"comment":"The paper's central motivation is reducing temporal jittering, but no temporal consistency metric (e.g., warping error, VTM, or frame-to-frame landmark stability) is reported; Table 1 reports only LSE-C/LSE-D and FID, and Table 3's temporal-attention ablation reports only LSE-C/LSE-D with the text saying it 'slightly reduced frame-to-frame consistency' without a measured quantity. Given that training is done on 12-frame clips and evaluation is on 8-second videos, the jitter-reduction claim is currently not directly supported. Please add a temporal metric or quantify the consistency gain.","section":"Sections 3.5 and 4 / jitter claim"}],"minor_comments":[{"comment":"The table has formatting errors and mismatched references: entries such as '4min27sDreamTalk' and '22.52.33' are merged, and the citations DreamTalk [27], SadTalker [51], and Hallo [47] do not match the reference list numbering used elsewhere.","section":"Table 2"},{"comment":"The text refers to 'the bias injection strategy (stated in 3.3.2)', but the subsection describing bias injection is 3.4.2.","section":"Section 3.4.1"},{"comment":"Line 10 defines xt as a fresh sample from N(sqrt(1-beta_{t-1}) x_{t-1}, beta_{t-1} I) after x_{t-1} has already been computed in line 9; this is notationally confusing and should be clarified or removed.","section":"Algorithm 1"},{"comment":"The repository link in the abstract is missing, and there are inconsistent capitalization/spelling variants such as 'Aniportrait' vs. 'AniPortrait', 'Hubert-base-ls960' vs. 'Hubert-Base-LS960', and 'V oxCeleb-HQ' vs. 'VoxCeleb-HQ'.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The central lip-sync claim is not supported by the authors' own Table 1, and the renderer ablation conflates Wav2Lip reference injection with the proposed method. The paper may become acceptable after a careful re-evaluation that either excludes Wav2Lip from the headline comparison or ablates the Wav2Lip reference in isolation, and after the Table 3/Table 4 discrepancy and the reversed metric direction in Section 4.2.1 are resolved. I would not recommend acceptance in the current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a real system, not a toy, and the 3DMM-conditioned diffusion transformer with biased attention phases is a sensible design. But the paper's central sentence — \"best lip-sync performance across the Same-Identity\" — is false as printed. Table 1 lists Wav2Lip at LSE-C 9.652 / LSE-D 5.656 and MoDiT at 7.428 / 7.645 on HDTF; Wav2Lip wins on both. Same pattern on VFHQ. And Wav2Lip is not just a baseline: Section 3.4.5 feeds Wav2Lip's output into the Reference UNet. So the claim is internally contradicted, and the margin over Hallo/DreamTalk may be inherited from the Wav2Lip reference rather than produced by MoDiT's own motion prediction.\n\nWhat's genuinely new: the hierarchical bias injection that switches from diagonal lip-focused attention to dispersed expression attention across diffusion phases, and the learned temporal attention offset. The idea of predicting 3DMM coefficients with a diffusion transformer and then using those to guide flow-based warping is a reasonable combination. The blink module with a learnable blink sequence is a small but sensible improvement over binary blink control. The ablations in Tables 3 and 4, inconsistent as they are, do show the components matter: removing 3DMM or temporal attention or the renderer reference degrades performance substantially. So there is real content here.\n\nSoft spots, in order of severity. First, the Table 1 contradiction. Either Wav2Lip should be excluded from the \"best\" claim with an explicit caveat, or the claim should be reworded to \"best among one-shot talking head methods.\" Second, the Wav2Lip-in-the-loop circularity: the evaluation uses SyncNet-family metrics (LSE-C/LSE-D) and the pipeline consumes Wav2Lip's output, so the reported lip-sync numbers partly reflect Wav2Lip's own optimization. The right ablation would toggle the Wav2Lip reference while keeping the Reference UNet; removing the whole unit doesn't separate the effects. Third, Table 3 and Table 4 report different values for \"w/o 3DMM source β0\" (5.365/9.431 vs 1.478/13.596), and the text says removing it gives \"low LSE-D, high LSE-C\" — the opposite of what the tables show. Fourth, no error bars, small training set (1k videos, 100 subjects), and FID is worse than several baselines, so the \"state-of-the-art comparable\" phrasing should be softened.\n\nWho it's for: people building diffusion-based talking head systems will find the bias injection and blink ideas worth trying. The evaluation needs major rework before publication, and the authors should release code. It deserves a serious referee, but I would not accept the current claims as written. If I were handling it, I'd send to review with a request for corrected tables, the Wav2Lip-reference ablation, and reworded claims.","headline":"A genuinely assembled diffusion-transformer system with a plausible blink module, but its headline lip-sync claim is contradicted by its own Table 1 because Wav2Lip — which is also inside the renderer — scores higher on both LSE metrics.","tokens_in":12678,"tokens_out":3153,"would_cite":false,"duration_ms":32737,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MoDiT achieves better same-identity lip sync, reduced jitter, and more natural blinks in talking-head videos by conditioning a diffusion transformer on 3DMM expression coefficients.","keywords":["talking head generation","3D morphable model","diffusion transformer","lip synchronization","temporal consistency","optical flow prediction","blink modeling"],"falsifier":"Measure MoDiT's LSE-C/LSE-D on a fixed audio-visual test set while replacing the Wav2Lip reference with a deliberately mismatched or blanked lip region; if scores do not drop substantially, the renderer is not actually relying on the reference as described, undercutting the claim that the pipeline inherits Wav2Lip's accuracy.","tokens_in":11597,"feed_emoji":"🗣️","tokens_out":8805,"duration_ms":79706,"temperature":0.7,"pith_summary":"This paper tries to establish that a diffusion transformer conditioned on 3D Morphable Model (3DMM) expression coefficients can generate audio-driven talking-head videos that are more temporally consistent, identity-preserving, and natural in blinking than existing GAN- and UNet-based methods. The core idea is that explicit 3D coefficients give the renderer spatial constraints, while a hierarchical denoising strategy with biased attention and revised temporal attention reduces frame-to-frame jitter. The authors report the best same-identity lip-sync scores on the HDTF and VFHQ datasets and SOTA-comparable results for cross-identity and out-of-distribution inputs. If true, the work would strengthen the case for using explicit 3D representations inside diffusion models for face animation.","feed_headline":"MoDiT achieves best same-identity lip sync on two benchmarks","feed_subtitle":"Combines 3D morphable coefficients with a diffusion transformer to cut jitter and preserve identity in audio-driven portraits.","key_machinery":"The central object is the sequence of 3DMM expression coefficients $\\beta$, which acts as the learned motion representation that the diffusion transformer directly denoises. The key mechanism is the bias injection strategy (Algorithm 1), which switches attention bias between a diagonal lip-focused mask at high noise and a dispersed expression mask at low noise, forcing the model to first lock onto lip motion and then refine full-face coherence. This, together with the revised temporal attention ($\\tilde{s}_t = s_t + f(z_t)$), carries the temporal-consistency claim; the 3DMM coefficients carry the identity-preservation claim by providing explicit geometry to the optical-flow renderer.","core_discovery":"MoDiT represents the 3D face as a sequence of expression coefficients $\\beta \\in \\mathbb{R}^{64}$ from the LSFM morphable model, and a diffusion transformer denoises this sequence conditioned on audio latents and the source image. A bias injection strategy applies a diagonal lip-focused attention bias during early high-noise denoising steps and a dispersed expression bias in later steps, while a revised temporal attention adds a learned latent variable to the attention scores. The denoised coefficients go through a mapping net that predicts head pose and eye blinks using a learnable blink sequence, and a renderer combines a Wav2Lip reference, a Reference UNet, and 3DMM-guided flow prediction to warp the source image. The paper claims this yields the best LSE-C/LSE-D lip-sync metrics on same-identity tests on HDTF and VFHQ and SOTA-comparable results otherwise.","pith_inferences":["A testable extension the paper leaves implicit: feeding the renderer a deliberately corrupted Wav2Lip reference should degrade lip-sync metrics if the pipeline truly depends on that reference; the degree of degradation would disentangle MoDiT's contribution from Wav2Lip's.","The bias injection schedule suggests that attention masking could be time-tuned in other diffusion tasks where early denoising should focus on a small but semantically critical region, such as image inpainting of a face.","The revised temporal attention with a learned latent variable may transfer to non-face motion generation, where sequence-level context matters but full attention is too expensive."],"forward_implications":["If the claim holds, audio-driven talking heads can be generated with less visible jitter, since temporal attention adaptively weights sequence context rather than treating all frames alike.","Explicit 3DMM conditioning should make the framework controllable: practitioners could edit the expression coefficients directly to change emotion or emphasis without retraining.","Because the renderer predicts optical flow from a known 3D geometry, the approach may extend more gracefully to large head poses than latent-only diffusion methods.","The learnable blink sequence provides a template for modeling other subtle, fast facial motions that deterministic regression tends to smooth away."],"supporting_citations":[{"why":"Blanz and Vetter's morphable model supplies the 3DMM identity and expression bases that parameterize the predicted motion coefficients.","marker":"[1]"},{"why":"Accurate 3D face reconstruction method used to extract 3DMM coefficients from training data and source images.","marker":"[5]"},{"why":"Pre-trained lip-reading model provides the lip-reading loss used in training the expression network.","marker":"[24]"},{"why":"Diffspeaker is the source of the bias injection strategy idea for phase-dependent attention in diffusion.","marker":"[26]"},{"why":"Wav2Lip gives the lip-synced reference input to the Reference UNet and serves as a strong baseline for comparison.","marker":"[28]"},{"why":"Hallo is a diffusion-based talking head baseline trained on a larger private dataset, used to contextualize the SOTA-comparable claims.","marker":"[46]"},{"why":"SadTalker is a 3DMM-based baseline that also provides the landmark loss and mapping-net design, used for comparison.","marker":"[49]"}],"fun_headline_variants":["Diffusion transformer tames talking head jitter","3D morphable coefficients boost lip sync consistency","MoDiT: 3DMM meets diffusion for stable avatars","New model cuts jitter, keeps identity in talking heads","Audio-driven faces stay stable with MoDiT"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the Wav2Lip reference provides an accurate enough lip-sync start that the renderer's optical flow can refine it; if that reference is misaligned, the entire pipeline inherits the misalignment, so the claimed lip-sync improvement may in part come from Wav2Lip rather than from MoDiT's 3DMM motion prediction.","fun_headline_variants_meta":{"raw":{"variants":["Diffusion transformer tames talking head jitter","3D morphable coefficients boost lip sync consistency","MoDiT: 3DMM meets diffusion for stable avatars","New model cuts jitter, keeps identity in talking heads","Audio-driven faces stay stable with MoDiT"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000199,"raw_usage":{"total_tokens":1385,"prompt_tokens":975,"completion_tokens":410,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":591,"completion_tokens_details":{"reasoning_tokens":332}},"tokens_in":591,"tokens_out":410,"duration_ms":4789,"temperature":1.0,"reasoning_tokens":332,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:32:49.037447+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure MoDiT's LSE-C/LSE-D on a fixed audio-visual test set while replacing the Wav2Lip reference with a deliberately mismatched or blanked lip region; if scores do not drop substantially, the renderer is not actually relying on the reference as described, undercutting the claim that the pipeline inherits Wav2Lip's accuracy.","supporting_citations":[{"cited_title":"A morphable model for the synthesis of 3d faces","cited_arxiv_id":null,"evidence_quote":"Blanz and Vetter's morphable model supplies the 3DMM identity and expression bases that parameterize the predicted motion coefficients."},{"cited_title":"Accurate 3d face reconstruction with weakly-supervised learning: From single image to image set","cited_arxiv_id":null,"evidence_quote":"Accurate 3D face reconstruction method used to extract 3DMM coefficients from training data and source images."},{"cited_title":"Training strategies for improved lip- reading","cited_arxiv_id":null,"evidence_quote":"Pre-trained lip-reading model provides the lip-reading loss used in training the expression network."},{"cited_title":"A lip sync expert is all you need for speech to lip generation in the wild","cited_arxiv_id":null,"evidence_quote":"Wav2Lip gives the lip-synced reference input to the Reference UNet and serves as a strong baseline for comparison."},{"cited_title":"Sadtalker: Learning realistic 3d motion coefficients for stylized audio- driven single image talking face animation","cited_arxiv_id":null,"evidence_quote":"SadTalker is a 3DMM-based baseline that also provides the landmark loss and mapping-net design, used for comparison."}],"review_version":1}