{"id":"a2b49650-ccd7-47f1-aa5a-abef4c5b7934","arxiv_id":"2504.18810","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Training a talking-face generator with an auxiliary uncertainty module that predicts its own pixel errors and matches error and uncertainty histograms improves image-quality metrics, though not lip-sync relative to Wav2Lip.","lead":"A talking-face video generation model called JULNet adds an uncertainty-learning branch that predicts where its own generated images will be wrong, and uses that signal to train the generator. The method improves image-quality metrics over several baselines, but its lip-sync metrics remain worse than Wav2Lip, contrary to the abstract.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's headline claim of superior audio-lip synchronization is contradicted by its own quantitative results: JULNet has worse LSE-D/LSE-C than Wav2Lip on both HDTF and MEAD (Tables 1 and 2), and Section 4.1.2 concedes this.","rationale":"I read the paper as claiming that joint uncertainty learning improves both fidelity and lip-sync relative to prior art. The fidelity part is supported by Tables 1 and 2 (PSNR, LPIPS, FID) and by the ablation in Table 3. The lip-sync part is not: on both datasets Wav2Lip beats JULNet on LSE-D and LSE-C, and the paper admits this in Section 4.1.2. This is not an outside-consensus disagreement but an internal inconsistency between the abstract/conclusion and the presented evidence. The reader's weakest assumption about the histogram KL term (Section 3.3) is a legitimate secondary concern, since the ablation in Table 4 shows only small gains from L_un2 and no error bars; however, the most load-bearing issue is the unsupported lip-sync superiority claim, because it is part of the central claim as stated. The proposed check of re-evaluating with the standard Wav2Lip SyncNet would confirm whether the metric ordering is robust; if it is, the claim must be revised. The verdict remains conditional: the uncertainty module may still be a useful contribution, but the paper should not be accepted with the current abstract and conclusion wording.","tokens_in":13498,"tokens_out":12236,"duration_ms":126520,"concrete_test":"Directly compare the LSE-D and LSE-C rows for JULNet and Wav2Lip in Tables 1 and 2. To guard against the paper's from-scratch SyncNet being non-standard, re-evaluate both methods with the official pre-trained Wav2Lip lip-sync expert on the same HDTF and MEAD test clips; if Wav2Lip still has lower LSE-D and higher LSE-C, the abstract and conclusion's audio-lip synchronization superiority claim is unsupported and must be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim, restated in the abstract and conclusion, is that JULNet 'outperforms previous methods in terms of generating high-fidelity talking face videos with audio-lip synchronization.' This is internally contradicted by the paper's own evaluation. In Table 1 (HDTF), Wav2Lip achieves LSE-D 6.792 and LSE-C 8.877, while JULNet achieves 7.024 and 8.208, worse on both lip-sync metrics. In Table 2 (MEAD), Wav2Lip achieves 7.655 and 7.902, while JULNet achieves 7.801 and 7.147, again worse on both. Section 4.1.2 explicitly says 'our method falls short compared to Wav2Lip' for audio-visual synchronization. Thus, as stated, the central claim is false; at most the evidence supports superior image-quality metrics (PSNR, LPIPS, FID) and improved sync relative to the no-uncertainty ablation, not superiority over prior methods in lip synchronization. This overclaim matters because the abstract's 'and' explicitly couples high fidelity with audio-lip synchronization as the basis for the method's advantage. The paper should either re-run evaluation with a fair, standard SyncNet or revise the claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes JULNet, an audio-driven person-generic talking face generation method built on a DINet-style deformation-inpainting generator with an added joint uncertainty learning module. After generating a frame, a network predicts a per-pixel error map (defined as the L1 distance to the ground truth) and an uncertainty map, trained with a heteroscedastic regression loss L_un1 (Eq. 4) and a histogram-based KL divergence L_un2 (Eq. 8) that matches the uncertainty distribution to the error distribution. The total training loss combines uncertainty, adversarial, perceptual, and lip-sync losses. Experiments on HDTF and MEAD compare JULNet against Wav2Lip, EAMM, VideoReTalking, DINet, and TalkLip, reporting PSNR, LPIPS, FID, LSE-D, and LSE-C. The paper reports the best image-quality metrics among the compared methods, mixed lip-sync metrics (worse than Wav2Lip on both datasets), and ablations showing gains from the uncertainty losses. A plug-in experiment on Wav2Lip suggests that the uncertainty module can transfer to another baseline.","tokens_in":13802,"tokens_out":8357,"duration_ms":76028,"significance":"If the reported performance holds, joint uncertainty learning is a simple, architecture-agnostic auxiliary loss that could improve the image quality of talking-face generators and possibly their robustness. The paper's strengths are the clear quantitative comparisons on two datasets, the ablation studies that isolate the contributions of L_un1 and L_un2, and the plug-in experiment on Wav2Lip, which is a convincing way to demonstrate the module's generality. However, the headline claim of superior audio-lip synchronization is contradicted by the paper's own LSE-D/LSE-C numbers, and several load-bearing technical details (the inference-time use of the uncertainty module, the exact form of the histogram KL, and the plug-in integration protocol) need clarification before the claims can be fully assessed.","major_comments":[{"comment":"The abstract and the conclusion state that JULNet achieves 'superior high-fidelity and audio-lip synchronization' compared to previous methods, but the quantitative results in Tables 1 and 2 contradict the lip-sync part. On HDTF, JULNet has LSE-D 7.024 and LSE-C 8.208 versus Wav2Lip's 6.792 and 8.877; on MEAD, JULNet has LSE-D 7.801 and LSE-C 7.147 versus Wav2Lip's 7.655 and 7.902. Section 4.1.2 explicitly concedes that 'our method falls short compared to Wav2Lip' on audio-visual synchronization. Since the abstract explicitly couples fidelity with lip-sync as the basis of the claimed advantage, this overclaim is load-bearing. The authors should either revise the claims to match the evidence (image quality improvements and sync improvements over the no-uncertainty ablation) or rerun the evaluation with a SyncNet setup that is directly comparable to Wav2Lip's, such as using the same pretrained sync expert.","section":"Abstract; Conclusion; Tables 1 and 2; Section 4.1.2"},{"comment":"The KL divergence direction is inconsistent and likely incorrect. Equation (5) writes D(P_σ(p))∥Q_ε(p), placing the uncertainty distribution as the left argument, while the text says the error distribution is the reference. Equation (8), however, computes Σ_j H_ε(j) log(H_ε(j)/H_σ(j)), which is KL(H_ε || H_σ), the opposite direction. Because KL is asymmetric, this inconsistency changes the optimization objective and the interpretation of the regularizer. Please correct Eq. (5) and unambiguously state which distribution is the reference.","section":"Section 3.3, Eq. (5) and Eq. (8)"},{"comment":"The role of the uncertainty module at inference is not specified. The error map is defined using the ground truth (Eq. 2), and Eq. (3) conditions the Laplacian on both predicted and ground-truth pixels, so it is unclear whether the uncertainty module can be executed at test time when ground truth is absent. The paper should state explicitly whether the uncertainty map is a training-only auxiliary output or is also produced for new videos, and if the latter, what input is used at inference. Without this, the claim that the model 'provides reliable uncertainty estimates for the generated content' is not verifiable.","section":"Section 3.2, Eq. (2); Fig. 2"},{"comment":"The plug-and-play generalization experiment on Wav2Lip is reported without any implementation details. The paper does not describe how the uncertainty module is attached to Wav2Lip, which parts of the network are trained, how the losses are combined, or which hyperparameters are used. The improvements in Table 5, including better LSE-D and LSE-C than the original Wav2Lip, are therefore impossible to reproduce or interpret. Please include the integration and training protocol, or explicitly frame the result as a preliminary proof-of-concept and list all required details.","section":"Section 4.3.3, Table 5"}],"minor_comments":[{"comment":"There are citation errors: Section 4.2.1 says 'Wav2Lip [43] and TalkLip [38]' but Wav2Lip is reference [38] and TalkLip is reference [43]; Section 4.0.2 also cites TalkLip as [47] instead of [43].","section":"Section 4.2.1; Section 4.0.2"},{"comment":"Equation (2) uses the same symbol p_i for both the predicted and ground-truth pixels; please use distinct notation, for example \\ hat{p}_i and p_i, consistently throughout the paper.","section":"Section 3.2, Eq. (2)"},{"comment":"Equation (7) is garbled in the typeset version: the denominator appears as '˝m_j=0 exp(w_j(ε_i))', which is illegible. Please provide a clean formula, e.g., a softmax normalization over the histogram bin weights, so that the histogram construction is unambiguous.","section":"Section 3.3, Eq. (7)"},{"comment":"The symbols ⊕ and … in Figure 2 are not explained in the caption; please label the concatenation operation and clarify which connections are omitted.","section":"Figure 2"},{"comment":"The sentence 'The source image is identical to the ground truth image, with the exception that it masks the lower-half mouth region' is confusing; please rephrase to make clear that the source image is a masked version of the ground truth.","section":"Section 3.1.1"},{"comment":"The explanation for the lower lip-sync scores is that the syncnet is trained from scratch, but the paper does not specify the training data, architecture, or training protocol for this syncnet; please provide these details.","section":"Section 4.1.2"},{"comment":"The keyword 'Taking Face Video Generation' is a typo; it should be 'Talking Face Video Generation'.","section":"Keywords"}],"recommendation":"major_revision","confidential_remarks":"The core idea is interesting and the ablations are supportive, but the abstract and conclusion overreach on lip-sync superiority is a serious issue that will draw criticism from readers. The plug-in experiment on Wav2Lip is potentially the strongest supporting evidence, so the authors should detail it and also correct the KL-direction error. I do not see grounds for rejection if the claims and technical descriptions are brought in line with the reported experiments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: the uncertainty-learning formulation is real and the ablations back it up, but the paper's central 'also better lip-sync' claim is contradicted by its own Tables 1 and 2. Treat the contribution as a quality regularizer, not a sync breakthrough.\n\nWhat's actually new: first use of error-map/uncertainty-map prediction with a differentiable histogram KL divergence in talking-face generation. The histogram trick (Eqs. 6-8) is a small but real technical addition that makes distribution matching end-to-end trainable. The generalization experiment (Table 5) showing Wav2Lip+uncertainty improves image quality is the strongest evidence; it directly demonstrates the plug-in value.\n\nWhat the paper does well: the ablations are systematic. Each loss component contributes, and the uncertainty details ablations (B, L_un1, L_un2) show both terms matter. That's more than many talking-face papers do. The writing is mostly clear, and the limitation section is honest about pose and background issues.\n\nSoft spots, in order of importance. First, the overclaim. The abstract and conclusion say the method achieves 'superior... audio-lip synchronization,' but LSE-D and LSE-C in both main tables are worse than Wav2Lip, and Section 4.1.2 admits it. That is not a minor wording issue; it misstates the empirical result. Second, the key hyperparameter B=11 is selected on the test set (the ablation is on HDTF test). No held-out validation set is mentioned. Third, no error bars, no code/data, and a few implementation details are vague (e.g., exactly how the SyncNet is trained when they say 'from scratch' vs. the off-the-shelf evaluation SyncNet; whether baselines are trained under the same data). These are fixable but currently reduce confidence.\n\nOn the central mechanism: the circularity concern doesn't land. Predicting the L1 error from generated vs. GT is a standard auxiliary task, and the KL matches histograms of the same data—this is part of the training objective, not a self-fulfilling metric.\n\nBottom line: for someone working on talking-face generation, this is a worthwhile read and a reasonable baseline. It deserves peer review after the claims are aligned with the numbers. I'd cite the plug-in result if I needed a regularizer. Slightly overstates itself, but the core is sound.","headline":"The uncertainty regularizer works for image quality, but the paper's lip-sync superiority claim is contradicted by its own tables.","tokens_in":14334,"tokens_out":2363,"would_cite":true,"duration_ms":23230,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that audio-driven talking-face video generation improves when the model is trained to predict its own per-pixel visual uncertainty—an error map and a matching uncertainty map—and shows that JULNet, built on this joint…","keywords":["talking face video generation","uncertainty learning","joint uncertainty","audio-lip synchronization","histogram KL matching","JULNet"],"falsifier":"Rerun the ablation in Table 4 over multiple random seeds; if removing $L_{\\mathrm{un2}}$ (the KL histogram term) yields no consistent degradation in PSNR, LPIPS, FID, or LSE metrics, or if replacing it with a simple moment-matching or variance penalty reproduces identical gains, the claim that histogram-based distribution matching is the active ingredient would be falsified.","tokens_in":13295,"feed_emoji":"🗣️","tokens_out":8657,"duration_ms":69811,"temperature":0.7,"pith_summary":"The paper tries to establish that a talking-face video generator can be made better and more reliable by teaching it to predict its own visual uncertainty. Its proposed network, JULNet, synthesizes a frame and then produces two maps: an error map showing the pixel-by-pixel difference from the ground truth, and an uncertainty map predicting where the model is likely to be wrong. A histogram-based KL divergence term forces the uncertainty distribution to match the error distribution, so that optimizing error and uncertainty together becomes a single training objective. The authors report that this joint uncertainty learning outperforms prior person-generic methods in image fidelity and audio-lip synchronization, and that the uncertainty module can be bolted onto an existing method such as Wav2Lip to improve it.","feed_headline":"Uncertainty-aware training sharpens talking-face generation","feed_subtitle":"By predicting where it may err and matching that prediction to actual errors, JULNet beats prior methods on quality and lip sync.","key_machinery":"The machinery is the joint uncertainty loss $L_{\\mathrm{un}} = L_{\\mathrm{un1}} + L_{\\mathrm{un2}}$. $L_{\\mathrm{un1}}$ is the standard heteroscedastic uncertainty-weighted regression loss: pixels with large predicted uncertainty $\\sigma_i$ contribute less to the error term, while the $\\tau_i$ sum prevents the model from inflating uncertainty everywhere. $L_{\\mathrm{un2}}$ is the distinctive piece: the paper approximates the error distribution and the predicted uncertainty distribution with a differentiable histogram, using soft bin assignments $w_j(\\epsilon_i) = \\lambda_1 \\exp(-(u_\\epsilon + \\alpha_j s_\\epsilon - \\epsilon_i)^2 / \\lambda_2)$, where $u_\\epsilon$ and $s_\\epsilon$ are the mean and standard deviation of the error and $\\alpha_m$ is a temperature-scale bin layout. It then computes the KL divergence between the two histograms, so that uncertainty is explicitly trained to have the same distribution as the error. This histogram tie makes the uncertainty representation directly related to visual error and is what carries the claimed improvement.","core_discovery":"JULNet is a person-generic audio-driven talking-face generator whose training loss includes a joint uncertainty term. After the generator produces an image $I_{gen}$, an uncertainty module computes an error map $\\epsilon_i = |\\hat{p}_i - p_i|$ (the L1 difference between generated and ground-truth pixels) and predicts an uncertainty map $\\sigma_i$, modeled as the scale of a Laplacian distribution. The uncertainty loss has two components: an uncertainty-weighted regression loss $L_{\\mathrm{un1}} = \\frac{1}{n}\\sum_i (\\epsilon_i / \\exp(\\tau_i) + \\tau_i)$, where $\\tau_i = \\log \\sigma_i$, and a KL matching loss $L_{\\mathrm{un2}}$ computed between differentiable histograms of the error and uncertainty values. The paper's central discovery is that jointly optimizing these two terms—forcing the uncertainty map's histogram to mirror the actual error histogram—improves PSNR, LPIPS, FID, and lip-sync metrics over prior methods, and that the same module improves Wav2Lip when attached to it.","pith_inferences":["Editorial extension: The differentiable-histogram KL scheme is not specific to talking faces; it could serve as a generic calibration regularizer for any generative model with access to pixel-level ground truth, such as inpainting or super-resolution.","Editorial extension: The predicted uncertainty map may have practical value as a failure detector: regions with high predicted uncertainty could be highlighted for human review or fed back into a second-stage refinement network.","Editorial extension: Since Wav2Lip still leads on LSE-C on HDTF, a direct further step would be to pre-train the sync expert on a larger dataset and then add the uncertainty module, testing whether the two improvements are complementary rather than overlapping.","Editorial extension: A testable hypothesis is that the KL histogram term mostly acts as a regularizer that smooths the uncertainty map; if true, a simpler variance-minimization or entropy penalty on the uncertainty map could reproduce the gains, which would simplify the method."],"forward_implications":["On HDTF and MEAD, JULNet reports the best PSNR, LPIPS, and FID among Wav2Lip, EAMM, VideoReTalking, DINet, and TalkLip, with the largest margins in FID (7.825 vs 8.437 on HDTF; 19.525 vs 23.350 on MEAD).","Adding the uncertainty module to Wav2Lip improves its PSNR from 31.836 to 32.108 and its FID from 9.137 to 6.996 on HDTF, indicating the module is a plug-and-play enhancement.","Ablations show each loss component matters: removing perception loss hurts realism, removing lip-sync loss hurts LSE metrics, and removing the uncertainty loss degrades both quality and sync (full model PSNR 32.266 vs 31.939 without the uncertainty loss).","The full uncertainty loss works best with 11 histogram bins, and both components $L_{\\mathrm{un1}}$ and $L_{\\mathrm{un2}}$ individually contribute, with $L_{\\mathrm{un2}}$ providing a smaller but consistent gain (PSNR 32.214 without it vs 32.266 with it).","The method's current limitations are dynamic scenes, changing illumination, moving backgrounds, and non-frontal head poses, where it can produce visible artifacts."],"supporting_citations":[{"why":"It supplies the uncertainty-weighted regression loss that $L_{\\mathrm{un1}}$ extends.","marker":"[22]"},{"why":"It justifies modeling the error distribution as Laplacian when the L1 loss is used for the error map.","marker":"[18]"},{"why":"It demonstrates in stereo matching that uncertainty is highly correlated with prediction errors, motivating the error-distribution matching approach.","marker":"[6]"},{"why":"It introduces Wav2Lip, the pioneering lip-sync baseline and the target method for the generalization experiment.","marker":"[38]"},{"why":"It provides the AdaAT deformation operation that JULNet's generator builds on, and serves as a comparison baseline.","marker":"[47]"},{"why":"It supplies the SyncNet used both for the lip-sync loss and for the LSE-D and LSE-C evaluation metrics.","marker":"[9]"},{"why":"It provides the HDTF dataset used for training and testing.","marker":"[48]"},{"why":"It provides the MEAD dataset used for training and testing.","marker":"[44]"}],"fun_headline_variants":["Uncertainty learning sharpens talking-face video","Predicting errors improves talking-face generation","Joint error-uncertainty boosts lip sync and quality","Talking-face video gains from uncertainty matching","Uncertainty-aware training enhances face video fidelity"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that the pixel-by-pixel difference between generated and real video frames faithfully captures how bad the output looks, and that pushing the predicted uncertainty map to have the same distribution as this difference teaches the network something it would not otherwise learn.","fun_headline_variants_meta":{"raw":{"variants":["Uncertainty learning sharpens talking-face video","Predicting errors improves talking-face generation","Joint error-uncertainty boosts lip sync and quality","Talking-face video gains from uncertainty matching","Uncertainty-aware training enhances face video fidelity"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000141,"raw_usage":{"total_tokens":1179,"prompt_tokens":975,"completion_tokens":204,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":591,"completion_tokens_details":{"reasoning_tokens":134}},"tokens_in":591,"tokens_out":204,"duration_ms":2617,"temperature":1.0,"reasoning_tokens":134,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:08:24.770761+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the ablation in Table 4 over multiple random seeds; if removing $L_{\\mathrm{un2}}$ (the KL histogram term) yields no consistent degradation in PSNR, LPIPS, FID, or LSE metrics, or if replacing it with a simple moment-matching or variance penalty reproduces identical gains, the claim that histogram-based distribution matching is the active ingredient would be falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It supplies the uncertainty-weighted regression loss that $L_{\\mathrm{un1}}$ extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It justifies modeling the error distribution as Laplacian when the L1 loss is used for the error map."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It demonstrates in stereo matching that uncertainty is highly correlated with prediction errors, motivating the error-distribution matching approach."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It provides the AdaAT deformation operation that JULNet's generator builds on, and serves as a comparison baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It provides the HDTF dataset used for training and testing."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It provides the MEAD dataset used for training and testing."}],"review_version":1}