{"id":"dd8e67a7-e26f-4b22-a172-5e41c856a550","arxiv_id":"2501.04586","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"IPTalker dubs videos by aligning audio with reference mouth images, warping them to match the target lip shape, and inpainting the result.","lead":"IPTalker is a new video dubbing system that uses attention to link driving audio to reference mouth images, then warps and inpaints those images to keep the speaker's identity while syncing lips. The authors report state-of-the-art scores on HDTF and VFHQ, but the evaluation has fairness gaps.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table I does not hold the reference-image budget constant: IPTalker consumes N reference images while all baselines are one-shot, so the claimed state-of-the-art margins may reflect test-time identity access rather than method superiority.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: comparing a multi-reference method against one-shot baselines cannot support a general state-of-the-art claim. The paper's own fine-tuning table demonstrates how much the metrics move when the model sees frames from the target identity's test video, which strongly suggests that reference-set composition is not a minor detail. Even if the architecture is plausible and the warping idea is reasonable, the empirical claim is the central contribution, and it is currently unsupported by a controlled comparison. I do not find a separate objection that would change the verdict: the use of SyncNet for both training and LSE-C/LSE-D evaluation is a known circularity in this literature and would deserve scrutiny, but the visual-quality metrics are independent of it and the reference-count confound is more fundamental. The manuscript also contains unresolved placeholders ('Figure ??'), typos, and incomplete references, and no code or checkpoints are provided, which further weakens verifiability but is secondary to the evaluation-design flaw. Under these conditions, rejection with an invitation to revise the evaluation protocol is appropriate.","tokens_in":12440,"tokens_out":3233,"duration_ms":34665,"concrete_test":"Re-run the Table I comparison with IPTalker restricted to N=1, using exactly the same single source frame (same crop, mask, and alignment) that each one-shot baseline receives. Also report N-sensitivity curves for N=1, 2, 4, 8, and include per-video standard errors. If the SSIM/LPIPS margin over DINet shrinks to within variance at N=1, or if the margin only persists when references are sampled from the target test clip, the SOTA claim is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central SOTA claim rests on Table I, but the evaluation does not control for the number of reference frames. IPTalker's alignment module (Sec. III-B) takes N reference mouth images and its warping module concatenates N reference features (Eq. 4), whereas every baseline is a one-shot method given a single source image. The paper never states N, nor whether the reference frames are drawn from the same test video whose frames are being reconstructed. If they are, the high SSIM/PSNR/LPIPS scores partly measure the availability of ground-truth-adjacent identity texture at inference time, not a fair algorithmic advantage. The identity fine-tuning experiment (Sec. IV-D) compounds this concern: after 200 steps on a split of the test video, SSIM rises from 0.6332 to 0.6724, showing that the reported metrics are highly sensitive to access to the target identity's own video. Therefore, the claim of 'consistently outperforming existing approaches' and establishing a new state of the art is not supported by an apples-to-apples comparison.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents IPTalker, a video dubbing framework composed of an audio-visual alignment module (transformer-based), a warping module using AdaIN, and an inpainting module with SPADE. The alignment module takes N reference mouth images and the driving audio to produce an identity-audio embedding; the warping module produces a flow to deform reference features; the inpainting module fills the masked mouth region. The model is trained with perception, GAN, and SyncNet-based lip-sync losses. Experiments on HDTF and VFHQ compare against recent one-shot methods (DreamTalk, MakeItTalk, Audio2Head, SadTalker, Hallo, AniPortrait, DINet) and report superior SSIM, PSNR, LPIPS, LSE-C, LSE-D, plus ablations and identity fine-tuning. The paper claims a new state of the art for identity-preserving video dubbing.","tokens_in":12705,"tokens_out":5095,"duration_ms":46788,"significance":"If the quantitative results are taken at face value, IPTalker would be a strong identity-preserving video dubbing method, and the component-wise ablations suggest the alignment, SPADE, and cross-modalities encoder each contribute. The paper explicitly includes a Limitations section and reports an identity fine-tuning study, which is transparent. However, the significance is conditional on a fair evaluation against matched reference budgets and independent lip-sync metrics; the current comparison does not establish the stated state of the art.","major_comments":[{"comment":"The central SOTA claim is not supported by a controlled comparison. IPTalker's alignment module takes N reference mouth images (Eq. 1) and the warping module concatenates N reference features (Eq. 4), whereas every baseline in Table I is one-shot, receiving a single source image. The value of N is never stated, and the paper does not say whether the reference frames are drawn from the same test video whose frames are being reconstructed. If they are, the large SSIM/PSNR/LPIPS margins partly measure access to ground-truth-adjacent appearance at inference time. The authors should either evaluate IPTalker with N=1, or add multi-reference baselines and an ablation over N.","section":"Sec. IV-D, Table I; Sec. III-B, Eq. (1); Sec. III-C, Eq. (4)"},{"comment":"The lip-sync evaluation is circular with the training objective. The model is trained with L_sync = E[(SyncNet(A,I_O)-1)^2], and the reported LSE-C/LSE-D metrics are derived from the same SyncNet (the paper says 'inspired by [1]', which uses SyncNet). Any improvement in LSE-C/LSE-D may be an artifact of directly optimizing the same network. To support the lip-sync claim, the authors should evaluate with at least one independent audio-visual synchronization model or a human study.","section":"Sec. III-E, Eq. (12) vs Sec. IV-B"},{"comment":"The identity fine-tuning experiment shows that a few hundred steps of training on a split of the test video raises SSIM from 0.6332 to 0.6724 and PSNR from 30.89 to 34.08. This indicates the reported metrics are highly sensitive to access to the target identity's own video. The paper must clarify whether Table I uses the fine-tuned or non-fine-tuned model; if fine-tuned, the comparison with non-fine-tuned baselines is unfair, and if not, the fine-tuning results reveal that the base model's advantage may shrink when the reference budget is matched.","section":"Sec. IV-D, Table II"},{"comment":"The experimental report lacks crucial details: the number N of reference images, training hyperparameters (epochs, learning rate, batch size, number of AVAU layers), and any error bars or significance tests over the 30 (HDTF) and 20 (VFHQ) test videos. Given the large variance typical of talking-head generation, the reported point estimates without variance do not support the 'consistently outperforms' claim.","section":"Sec. IV-A, Sec. IV-D"}],"minor_comments":[{"comment":"The reference to 'Figure ??' is unresolved; the figure for the warping network architecture is missing.","section":"Sec. III-C"},{"comment":"The summation index i in the perception loss is overloaded with the number of reference images N; using L or a layer index would avoid confusion.","section":"Sec. III-E, Eq. (8)"},{"comment":"Typos: 'btween' should be 'between', and 'pixl-level' should be 'pixel-level'.","section":"Sec. IV-B"},{"comment":"The phrase 'conpensate the occlusion regions' should be 'compensate for the occlusion regions'.","section":"Conclusion"},{"comment":"Ground Truth PSNR is listed as N/A; the authors should explain, as PSNR can be computed for a perfect reconstruction.","section":"Table I"},{"comment":"The sentence that each test video is 'divided into two segments on average' is ambiguous; please specify the exact split.","section":"Sec. IV-D"},{"comment":"The Limitations paragraph raises a 'floating mouth' issue that is not analyzed in the experiments; consider discussing this in the main evaluation.","section":"Limitations"}],"recommendation":"major_revision","confidential_remarks":"The paper has a plausible architecture but the evaluation does not match the claimed state of the art. The comparison is unfair (multiple references vs one-shot baselines), and the lip-sync metric is circular. I would be willing to reconsider after a major revision that controls the reference budget, reports N, adds an independent lip-sync metric, and provides statistical significance. If the authors cannot provide N=1 results, a reject may be warranted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is an incremental but coherent dubbing architecture, and the mechanics are laid out cleanly. The SOTA claim does not survive a close read. The stress-test note is right: IPTalker consumes N reference mouth images in both the alignment module (Eq. 1-3) and the warping module (Eq. 4), while the Table I baselines are one-shot. The paper never states N, never says where the reference frames come from, and never controls for whether they are from the same test video. If they are, the SSIM/PSNR/LPIPS margins partly measure access to ground-truth-adjacent identity texture at inference time, not a fair algorithmic advantage. The identity fine-tuning experiment makes this concrete: 200 steps on a split of the same test video lifts SSIM from 0.6332 to 0.6724 and PSNR from 30.89 to 34.08. That is test-time identity access, and it muddies the main table.\n\nWhat is actually new is the Audio-Visual Alignment Unit: cross-attention between audio and N reference mouth crops, followed by a weighted correspondence embedding injected into an AdaIN-based warping network. That is a legitimate extension of DINet/FOMM/PIRenderer-style deformation, and the ablation study shows each component does something. The math is straightforward and the writing is clear. The limitations paragraph about a floating mouth is honest.\n\nSoft spots, in order of severity. First, the unfair comparison already described. Second, the lip-sync metric is circular: SyncNet appears in the training loss (Eq. 12) and in the reported LSE-C/LSE-D. That is common in this field, and the visual metrics are independent, so I would call it a real weakness but not disqualifying on its own. Third, there are no error bars or significance tests, and a few figure references are left as \"Figure ??\", which suggests the manuscript is not fully polished. Fourth, no code or data is released, so the numbers cannot be independently checked.\n\nWho is this for? People working on talking head synthesis and dubbing evaluation. The paper is a useful data point on using multiple reference images for identity preservation, but the current evaluation makes the headline claim unverifiable. I would send it to peer review rather than desk reject, because the idea is worth testing properly. The referee instructions should require a matched-reference-budget comparison, a statement of N and reference-frame provenance, and error bars. If those come back clean, the paper has a place; if not, it is an incremental contribution with an unsupported SOTA claim.","headline":"The architecture is a plausible incremental step, but the SOTA claim is not supported because Table I compares multi-reference IPTalker against one-shot baselines and the paper never isolates the reference budget.","tokens_in":733,"tokens_out":1006,"would_cite":false,"duration_ms":51977,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"IPTalker synthesizes dubbed video by aligning driving audio with reference mouth images in a transformer, then warping those images to the target mouth shape, and the paper shows this preserves the speaker's identity better than existing…","keywords":["video dubbing","identity preservation","motion warping","audio-visual alignment","talking head generation","lip synchronization","transformer attention","image inpainting"],"falsifier":"Run IPTalker with exactly one reference image on the same HDTF/VFHQ test videos used in Table I, measuring SSIM, PSNR, LPIPS, and LSE-C/LSE-D; if the metrics fall to the level of DINet or below, the claimed superiority is an artifact of the number of reference images rather than the alignment-warping mechanism.","tokens_in":12262,"feed_emoji":"🎬","tokens_out":4556,"duration_ms":41762,"temperature":0.7,"pith_summary":"This paper proposes IPTalker, a video dubbing framework that keeps the reference speaker's identity while matching lip movements to a new audio track. Instead of generating mouth pixels from scratch, it first learns which reference mouth image best corresponds to the driving audio, then spatially warps that image to the target mouth configuration, and finally inpaints any occluded region. The authors claim this approach produces high-resolution, temporally consistent dubbed videos with realistic mouth textures, and they report it outperforms prior open dubbing methods on both the HDTF and VFHQ datasets. The reason this matters is that previous dubbing models tend to trade lip-sync accuracy against identity fidelity, and IPTalker is designed to preserve both.","feed_headline":"Warp a reference mouth to the audio to dub video without losing identity","feed_subtitle":"IPTalker's transformer alignment and motion warping beats one-shot dubbing on HDTF and VFHQ quality metrics","key_machinery":"The key machinery is the Audio-Visual Alignment Unit (AVAU), a transformer block that treats audio and reference mouth embeddings as queries and keys to learn which reference mouth image best matches the target lip shape, alongside the warping module that uses the resulting correspondence embedding to modulate an encoder-decoder via AdaIN and produce a motion flow for spatial deformation. The SPADE decoder then inpaints the masked mouth region, and a Gaussian-smoothed convex-hull mask blends the result back into the frame. Together these components replace direct pixel generation from audio with a deform-and-inpaint scheme that preserves the reference texture.","core_discovery":"The paper's central claim is that explicit audio-visual correspondence learning, followed by motion warping of reference mouth images, is sufficient to achieve state-of-the-art video dubbing with strong identity retention. Given a source video and driving audio, IPTalker extracts embeddings from N reference lower-face images and the audio, uses stacked Audio-Visual Alignment Units and a cross-modalities encoder to produce an identity-audio correspondence embedding, then predicts a motion flow that warps reference features to the audio-driven mouth shape. A SPADE-based inpainting decoder fills the masked mouth region while a Gaussian-smoothed convex-hull mask pastes the generated face back onto the original frame. On quantitative metrics SSIM, PSNR, LPIPS, LSE-C, and LSE-D, and in qualitative comparisons, the paper reports IPTalker achieves the best visual quality among compared methods on both HDTF and VFHQ, with best lip-sync on HDTF and slightly lower LSE-D on VFHQ attributed to training on front-facing HDTF data only. The paper also acknowledges that because deformation is confined to the mouth region, the generated mouth can appear to 'float' relative to surrounding facial features when angle or position is slightly off.","pith_inferences":["Editorial: The paper compares IPTalker, which uses N reference mouth images, against one-shot baselines that see a single source image; this asymmetry may inflate identity-preservation margins, and a fair head-to-head would restrict IPTalker to one reference.","Editorial: The same deform-and-inpaint recipe could be applied to other audio-driven portrait editing tasks, such as singing voice dubbing or emotion-transferred speech, where reference texture fidelity matters.","Editorial: Since the method only warps the mouth region, its success suggests that audio mainly constrains lower-face geometry while other identity attributes can be carried over spatially; testing on profile views may expose where warping fails.","Editorial: A natural testable extension is to report IPTalker's performance with N=1, N=2, and N=5 references on the same test set to quantify how much of the reported advantage comes from the number of references."],"forward_implications":["If IPTalker's claim holds, video dubbing systems can keep the original speaker's teeth, tongue, lip color, and skin texture instead of synthesizing generic mouth interiors.","The warping-based pipeline should yield temporally stable output because most pixels in consecutive generated frames come from the same reference images rather than being regenerated per frame.","Identity fine-tuning on a short segment of a target video further improves visual quality, suggesting a practical workflow of per-identity adaptation for production use.","The alignment-by-correspondence design also provides a principled way to fuse audio with multiple reference views, which could extend to multi-view talking-head synthesis."],"supporting_citations":[{"why":"Supplies the lip-sync loss and SyncNet-based evaluation used to train and measure IPTalker's audio-visual synchronization.","marker":"[1]"},{"why":"HDTF dataset is one of the two test beds for the quantitative comparisons and identity fine-tuning experiments.","marker":"[11]"},{"why":"VFHQ dataset is the second test bed, with profile faces that stress the method's robustness.","marker":"[10]"},{"why":"Establishes the self-attention and cross-attention machinery on which the Audio-Visual Alignment Unit is built.","marker":"[15]"},{"why":"DINet is the strongest baseline and the source of the two-scale perception loss that IPTalker adopts.","marker":"[16]"},{"why":"AdaIN is the mechanism that injects audio-visual alignment features into the warping network before each layer.","marker":"[19]"},{"why":"SPADE normalization is used in the inpainting decoder to preserve semantic alignment with source facial features.","marker":"[22]"},{"why":"Mediapipe facial landmarks are used to compute convex-hull masks for the mouth region and the full face.","marker":"[21]"}],"fun_headline_variants":["Motion warping dubbing preserves identity and lip-sync","Audio-driven dubbing that warps mouth, not identity","IPTalker motion warping keeps dub identity intact","Warp reference mouth to audio, retain identity","New dubbing method warps mouth, preserves face identity"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that giving IPTalker N reference mouth images while giving each baseline only one source image is a fair comparison for the state-of-the-art claim; with a single reference image, IPTalker's reported identity-preservation and quality margins could shrink or disappear.","fun_headline_variants_meta":{"raw":{"variants":["Motion warping dubbing preserves identity and lip-sync","Audio-driven dubbing that warps mouth, not identity","IPTalker motion warping keeps dub identity intact","Warp reference mouth to audio, retain identity","New dubbing method warps mouth, preserves face identity"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000562,"raw_usage":{"total_tokens":2718,"prompt_tokens":1042,"completion_tokens":1676,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":658,"completion_tokens_details":{"reasoning_tokens":1612}},"tokens_in":658,"tokens_out":1676,"duration_ms":13311,"temperature":1.0,"reasoning_tokens":1612,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:28:43.614466+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run IPTalker with exactly one reference image on the same HDTF/VFHQ test videos used in Table I, measuring SSIM, PSNR, LPIPS, and LSE-C/LSE-D; if the metrics fall to the level of DINet or below, the claimed superiority is an artifact of the number of reference images rather than the alignment-warping mechanism.","supporting_citations":[{"cited_title":"A lip sync expert is all you need for speech to lip generation in the wild,","cited_arxiv_id":null,"evidence_quote":"Supplies the lip-sync loss and SyncNet-based evaluation used to train and measure IPTalker's audio-visual synchronization."},{"cited_title":"Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset,","cited_arxiv_id":null,"evidence_quote":"HDTF dataset is one of the two test beds for the quantitative comparisons and identity fine-tuning experiments."},{"cited_title":"Vfhq: A high-quality dataset and benchmark for video face super-resolution,","cited_arxiv_id":null,"evidence_quote":"VFHQ dataset is the second test bed, with profile faces that stress the method's robustness."},{"cited_title":"Dinet: Deformation inpainting network for realistic face visually dubbing on high resolution video,","cited_arxiv_id":null,"evidence_quote":"DINet is the strongest baseline and the source of the two-scale perception loss that IPTalker adopts."},{"cited_title":"Semantic image synthesis with spatially-adaptive normalization,","cited_arxiv_id":null,"evidence_quote":"SPADE normalization is used in the inpainting decoder to preserve semantic alignment with source facial features."}],"review_version":1}