{"id":"751f5920-6a3e-4090-9099-2ab7f89e348a","arxiv_id":"2505.03603","paper_version":5,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":9,"one_line_summary":"PAHA improves audio-driven avatar video generation by re-weighting training loss toward hands and face and by adding audio-video consistency classifiers during inference.","lead":"The paper introduces a system that turns a still picture of a person and a speech recording into a video where the person talks and gestures. The authors report better quality and lip-sync than earlier models, and they built a new Chinese news anchor dataset.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The PCE classifiers are trained to separate real videos from UniVDM-generated videos, not to detect audio-video mismatch, so guidance gains on BAS/Sync-C could reflect artifact suppression or test-set-tuned hyperparameters rather than true alignment.","rationale":"The reader's verdict is CONDITIONAL, and I agree with the weakest-assumption analysis. I single out the PCE labeling confound because the paper's novelty claim is audio-motion alignment, and PCE is the only component explicitly aimed at alignment. PAR alone is a spatial loss-weighting scheme that plausibly improves regional visual quality; the risky step is the claim that gradients from a real-versus-generated classifier improve synchronization. The binary cross-entropy loss (Eq. 7) does not reward the classifier for detecting mismatched audio-video pairs; it rewards any distinction between real videos and videos produced by the fixed UniVDM. Since the same generator is used at inference, the guidance can be viewed as a learned, region-masked realism correction on the generator's own manifold. This can improve perceptual metrics without establishing a causal link to audio. The reported BAS and Sync-C gains are suggestive, but they are not accompanied by confidence intervals, and the guidance hyperparameters were selected on the test set, so the wording significantly outperforms is not yet statistically supported. The proposed real-versus-mismatched classification test directly isolates whether the classifier learned correspondence; if it fails, the PCE contribution is unverified rather than disproven, and the paper should remain CONDITIONAL pending that evidence. If it passes, the central claim is materially strengthened. I therefore recommend keeping the reader's CONDITIONAL verdict.","tokens_in":17896,"tokens_out":8171,"duration_ms":85014,"concrete_test":"Take the trained face and non-face classifiers and evaluate them on real test videos paired with (i) their original audio and (ii) audio from a different clip of the same speaker, temporally aligned by the same frame count, across the same noise levels t used in the first 15 guided DDIM steps. If the classifiers cannot distinguish matched from mismatched real audio-video pairs at substantially above chance accuracy, then they have not learned audio-video correspondence; in that case, re-run Tables 1 and 3 with negative samples generated from a different UniVDM checkpoint and with lambda weights chosen on a validation split, and check whether the BAS and Sync-C gains persist. If the classifiers do distinguish matched from mismatched pairs and the gains persist, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that PAHA significantly outperforms prior methods in audio-motion alignment rests on the PCE classifier guidance (Sec. 4.1.3). The classifier's training labels are real versus generated, not synchronized versus mismatched: positives are ground-truth videos, while negatives are videos generated by the same UniVDM checkpoint (60k steps) conditioned on the same audio (Sec. 4.1.3, Appendix B.3.2). Such a classifier can minimize Eq. (7) by exploiting any perceptual difference between real and generated videos, including local sharpness, temporal smoothness, identity drift, or other generator-specific artifacts, without ever learning audio-video correspondence. At inference, the resulting gradients are masked to face and non-face regions (Eqs. 8-9), so they can improve regional realism and lower FGD/FVD even if true alignment is unchanged. The only direct alignment evidence is BAS and Sync-C, but those numbers are computed on the same PATS and CNAS test sets used to select lambda_nf, lambda_f, lambda_diff, and the guidance-time rate (Sec. 5.4, Appendix D), and no error bars or significance tests are reported. The paper's own ablations show classifier guidance slightly degrades FVD while improving alignment metrics; that pattern is consistent with a real tradeoff, but without a test that the classifier actually detects mismatched audio-video pairs, the alignment improvement remains confounded.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript proposes PAHA, an end-to-end latent diffusion framework for audio-driven upper-body avatar video generation. The backbone, UniVDM, is a 3D-UNet that processes the reference image and noisy video jointly without a separate reference encoder. PAR re-weights the denoising loss spatially using pose-keypoint confidence masks for hands, face, and body. PCE trains two binary classifiers (face and non-face) to distinguish real videos from videos generated by UniVDM, and uses their gradients during inference via Sequential Guidance or Differential Guidance. The paper also introduces CNAS, a Chinese news-anchor co-speech dataset, and reports quantitative results on PATS and CNAS against SDT, ANGIE, MM-Diffusion, and S2G, plus a small user study.","tokens_in":18219,"tokens_out":6966,"duration_ms":69913,"significance":"If the results hold, the paper makes a useful engineering contribution: it demonstrates that spatially re-weighting a video diffusion loss and guiding inference with regional classifiers can improve local quality and alignment metrics over several published baselines, and CNAS addresses a real gap in non-English co-speech data. Strengths include the unified-backbone design, the explicit regional-reweighting mechanism, comparisons with four baselines on two datasets, and the use of externally defined metrics such as FVD, BAS, FGD, Sync-C, and Div. The central limitation is evidential: the PCE classifier is trained on real-vs-generated labels rather than on audio-video synchronization labels, and the guidance hyperparameters are selected on the same test sets used for the headline numbers. These issues, together with the absence of error bars and significance tests, make the claim of significant outperformance currently under-supported.","major_comments":[{"comment":"The PCE classifiers are trained with binary cross-entropy on positive samples from ground-truth videos and negative samples generated by the same UniVDM checkpoint (60k steps) that they later guide. This supervision does not encode audio-video correspondence: a classifier can minimize Eq. (7) by detecting any perceptual difference between real and generated videos, including local sharpness, temporal smoothness, identity drift, or generator-specific artifacts, without ever learning whether motion is synchronized with audio. The reported BAS and Sync-C gains are therefore confounded as evidence for audio-motion alignment. I ask for a direct test of the classifier's alignment competence, for example evaluation on videos with shifted or replaced audio, reporting classification accuracy or AUC, and a comparison of guidance gradients for matched versus mismatched audio-video pairs. Alternatively, the negative samples should be constructed with mismatched audio rather than with generated videos. Without such evidence, the central claim that PCE enhances audio-video alignment is not established.","section":"Sec. 4.1.3, Eq. (7), Appendix B.3.2"},{"comment":"All quantitative tables report single-run metrics without error bars, confidence intervals, or significance tests, and the guidance weights lambda_non-face, lambda_face, and lambda_diff, together with the guidance-time rate, are selected by sweeping values on the PATS and CNAS test sets and choosing the combination achieving optimal performance. Because the same test sets are then used for the headline comparison in Table 1, the reported improvements may be optimistic, and several differences are small in absolute terms, for example FVD 2052.95 for PAHA-SG versus 2048.75 for PAHA-DG. Please add multiple seeds with mean and standard deviation, perform significance tests against the baselines, and select hyperparameters on a validation split or otherwise report the selection protocol explicitly before quoting final test-set numbers.","section":"Secs. 5.1, 5.4, Appendix D, Tables 1-3"},{"comment":"The user study uses 16 participants on 10 videos and reports only raw preference percentages. The instructions explicitly ask participants to ignore texture and facial expressions during motion evaluations, which can bias the synchrony and overall-quality ratings toward motion-based criteria and away from visual quality. No significance test, confidence interval, or inter-rater agreement is reported. Because the abstract and conclusion rely on the user study as evidence of significant outperformance, please add a statistical analysis, for example binomial or Wilcoxon tests with confidence intervals, and use neutral instructions that do not direct participants to discount specific quality dimensions, or at minimum discuss how the stated instructions affect the scope of the conclusions.","section":"Sec. 5.3, Table 4"},{"comment":"ANGIE is listed as a baseline but has no CNAS results in Table 1, and no explanation is provided. More importantly, the baseline fine-tuning protocol is described only as initialized with official weights and fine-tuned on the PATS and CNAS datasets; the reader cannot tell whether all baselines receive the same amount of training data, the same number of steps, or the same evaluation protocol. Please specify the fine-tuning setup in detail and make the evaluation code and data available, since the abstract promises release upon acceptance but no repository or data-access statement is included in the current manuscript.","section":"Sec. 5.1, Table 1"}],"minor_comments":[{"comment":"The Div value reported for PAHA-SG at 50% guidance in Table 2 is 99.604, but Tables 1 and 3 report 100.917 for what appears to be the same configuration; please reconcile this discrepancy.","section":"Tables 2 and 3"},{"comment":"In Eq. (7), the symbol y is described as the final predicted label, which is confusing because the classifier output is the synchronization score s; please rename the ground-truth label to y_true or similar.","section":"Eq. (7)"},{"comment":"The text says that 1,473 valid clips are obtained, while Appendix C states 1,473 valid clips per news anchor with 5 identity IDs; please clarify whether the count is per identity or the total number of clips in CNAS.","section":"Appendix C"},{"comment":"The paper repeatedly calls CNAS the first public Chinese News Anchor Speech dataset, but no dataset URL or release mechanism is provided in the manuscript; please add a data-availability statement.","section":"Sec. 4.3 and Abstract"},{"comment":"The discussion of the guidance-time rate says the best balance occurs from 0% to 50% guided steps and that further increases reduce performance, but FVD actually degrades monotonically while BAS and Sync-C improve; please state the criterion used to define the best balance.","section":"Sec. 5.4, Table 2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a competent systems contribution with a plausible mechanism, but the evaluation does not yet support the headline alignment claim. I would ask the editor to require the additional experiments described in major comments 1 through 3 before considering acceptance; the lack of code and data access is also a substantial barrier for this type of empirical contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper is a real contribution — a unified diffusion backbone for audio-driven upper-body animation, plus a confidence-weighted regional loss and a self-distilled classifier guidance scheme — but the evidence for the central alignment claim is softer than the abstract suggests. I'd send it to reviewers, not desk-reject, but I'd expect another round on the evaluation.\n\nWhat's genuinely new: UniVDM avoids the ControlNet-style reference encoder by fusing the reference image into the temporal dimension; PAR's confidence-weighted mask is simple and effective; PCE's two regional classifiers with Sequential and Differential guidance is a sensible way to inject alignment gradients at inference. The ablations are informative: removing PAR hurts FVD, the face classifier changes Sync-C, the non-face classifier changes FGD/BAS — the components behave as intended. The CNAS dataset is a useful resource for Chinese co-speech research.\n\nSoft spots, in order. First, the PCE classifiers are trained to distinguish real videos from UniVDM-generated videos, not matched from mismatched audio-video pairs. So the classifier could be rewarding artifact suppression rather than true audio-motion correspondence. The paper claims it learns 'the temporal correlation between spectral energy variations and localized character motion' but gives no sanity check that it responds to actual desynchronization. That is the load-bearing weakness. Second, guidance hyperparameters are chosen by running on the test set (Appendix D), and all metrics are single-run without error bars or significance tests. BAS improves from 0.1227 to 0.1561 and Sync-C from 5.29 to 6.18; those look like real gains, but I cannot tell how robust they are. Third, minor: the CNAS description contradicts itself (1,473 clips total vs per anchor), and the user study is small with instructions that may bias motion evaluation. Fourth, code and dataset are promised but not provided.\n\nNone of this sinks the paper. The central mechanism is plausible, the ablations support the design, and the external metrics (BAS, Sync-C, FGD, FVD) are independent of the classifier's training objective. But 'significantly outperforms' is overclaimed until the artifact-vs-alignment confound is addressed. I would ask the authors to test the classifier on deliberately mismatched pairs, report error bars, and pre-specify hyperparameter selection.\n\nWho should read it: people working on co-speech gesture video generation and classifier-guided diffusion. It deserves a serious referee; I would conditionally accept after the above is addressed.","headline":"A genuine method contribution with a plausible alignment mechanism, but the central 'significantly outperforms' claim currently rests on evidence that is still confounded and unreleased.","tokens_in":18766,"tokens_out":2891,"would_cite":true,"duration_ms":28596,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PAHA claims that splitting supervision by body part into loss re-weighting and classifier guidance makes a single diffusion model beat prior methods on avatar video quality and audio-motion alignment.","keywords":["audio-driven avatar generation","video diffusion model","parts-aware re-weighting","classifier guidance","co-speech gesture generation","audio-motion alignment","Chinese news anchor dataset","upper-body human animation"],"falsifier":"Retrain only the PCE classifiers with negative samples generated by a different, independently trained generator (or by real videos with time-shifted audio), keep the rest of PAHA fixed, and rerun the evaluation. If the FGD, BAS, and Sync-C gains vanish or reverse, the guidance was exploiting artifacts of the fixed 60k-step generator rather than learning general audio-motion correspondence.","tokens_in":17677,"feed_emoji":"🗣️","tokens_out":9912,"duration_ms":94465,"temperature":0.7,"pith_summary":"The paper aims to establish that audio-driven upper-body avatar video generation benefits from treating supervision by body part instead of as a whole-frame task. It introduces PAHA, an end-to-end video diffusion framework, with Parts-Aware Re-weighting (PAR), which redirects training loss toward the hands, face, and body according to pose-keypoint confidence, and Parts Consistency Enhancement (PCE), which trains face and non-face audio-video classifiers whose gradients steer generation at inference. The authors report that PAHA beats existing co-speech gesture generators on video quality, diversity, and audio-motion alignment on two benchmark datasets and in user studies, while avoiding intermediate pose, mesh, or landmark representations. They also build CNAS, described as the first public Chinese news-anchor speech dataset, to validate the method in a Chinese broadcast setting.","feed_headline":"Training by body part makes avatars match speech better","feed_subtitle":"PAHA reweights losses by pose confidence and uses audio-video classifiers to pull face and body motion into sync with speech.","key_machinery":"The central mechanism is a pair of diffusion-based regional audio-video classifiers, one trained on face regions and one on non-face regions, that output a synchronization score and provide gradient directions during sampling. The classifiers are trained with binary cross-entropy to tell real audio-video pairs from negative pairs produced by the already-trained generator, using a masked face/non-face split so each learns its own region's temporal correlation with the audio. Their gradients, masked and scaled by guidance weights, are applied to noised latents either sequentially (SG) or with a differential correction term (DG). PAR is the training-side companion: it converts pose-keypoint confidence scores into a Gaussian-smoothed, region-weighted loss mask so the diffusion model concentrates learning on hands and face.","core_discovery":"The central claim is that a single end-to-end diffusion model can simultaneously improve local visual quality and synchronization with speech if supervision is split by body region. During training, PAR builds a spatial loss mask from pose-keypoint confidence scores, drawing weighted circles around reliable hand and face keypoints and a weighted rectangle around the body, then amplifies the diffusion loss in those areas. During inference, PCE supplies gradient guidance from two masked classifiers, one for the face and one for the rest of the body, so that the denoising process is pulled toward audio-consistent motion in each region; a differential variant tries to stop each classifier's guidance from degrading the other region. On the four-talker benchmark and the new Chinese dataset, PAHA-SG and PAHA-DG report better FVD, BAS, Sync-C, FGD, and diversity than the evaluated baselines, with DG favored in quality and user preference.","pith_inferences":["If the PCE guidance mechanism is as general as the paper suggests, the same classifiers should be usable to steer other video-diffusion backbones without retraining the backbone, since they act on noised latents and audio; a direct cross-generator transfer test would check this.","PAR could be extended to finer regions such as eyes, teeth, or individual fingers, and to expression-specific areas; the paper's own limitations section notes that its face/hands/body split is still too coarse for complex expressions and overlapping gestures.","Because CNAS is small (five identities, 1,473 clips), a natural next test is cross-lingual and cross-identity transfer: train on Chinese data and evaluate on English data, and vice versa, to see whether PAHA's gains are language- or speaker-specific.","Since PAR reweighting relies on pose-estimator confidence, its benefit is capped by pose-estimator quality; making the weighting robust to low-confidence or occluded keypoints would be a testable improvement."],"forward_implications":["End-to-end generation removes dependence on intermediate 2D/3D pose, mesh, landmark, or optical-flow representations, reducing inference cost compared with multi-stage co-speech gesture pipelines.","The ablation results attribute a large share of the visual-quality gain to PAR: removing it raises FVD by about 93 points and lowers diversity by about 11 on the four-talker benchmark.","The two PCE classifiers are complementary: the face classifier carries most of the lip-sync improvement, while the non-face classifier carries most of the gesture improvement (without it, FGD rises about 19.5% and BAS falls about 16.6%).","Guidance strength and guidance duration are tunable: increasing the fraction of guided denoising steps improves alignment metrics but worsens FVD, with 50% of steps giving the reported best balance, and DG trades extra time for better quality than SG.","The CNAS dataset enables validation of co-speech gesture methods on Chinese news-anchor speech, a language and broadcast style not covered by the prior English benchmark."],"supporting_citations":[{"why":"Supplies the primary co-speech gesture video dataset and the four-talker training/evaluation configuration used for the backbone and classifiers.","marker":"[1]"},{"why":"Supplies the pre-trained latent video diffusion model from which the unified backbone is initialized and whose 3D-UNet structure is adopted.","marker":"[2]"},{"why":"Supplies the self-distillation idea of using the trained generator to produce negative samples for classifier training.","marker":"[3]"},{"why":"Defines the state-of-the-art co-speech gesture video baseline and the training subset that PAHA reuses for fair comparison.","marker":"[10]"},{"why":"Provides the hand/face/body region decomposition that PAR's Awareness Areas are based on.","marker":"[25]"},{"why":"Provides the wav2vec audio features that the model's audio attention layers condition on.","marker":"[33]"},{"why":"Supplies the DDIM sampler used both to generate negative classifier samples and to run 30-step inference.","marker":"[36]"},{"why":"Supplies the Fréchet Video Distance metric used to evaluate overall video quality.","marker":"[41]"},{"why":"Supplies the transformer encoder architecture used by the PCE classifiers for audio-video interaction.","marker":"[42]"}],"fun_headline_variants":["Split avatars by body part to improve speech sync","Body-part-aware training sharpens avatar speech alignment","Avatar motion syncs better with part-specific audio guidance","Part-wise training boosts avatar audio-visual alignment"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the face and non-face classifiers, trained partly on videos produced by the same generator they later steer, learn general audio-motion correspondence rather than a way to suppress that generator's characteristic errors, so their guidance improves real alignment for unseen inputs.","fun_headline_variants_meta":{"raw":{"variants":["Split avatars by body part to improve speech sync","Body-part-aware training sharpens avatar speech alignment","Avatar motion syncs better with part-specific audio guidance","Part-wise training boosts avatar audio-visual alignment"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000635,"raw_usage":{"total_tokens":2934,"prompt_tokens":959,"completion_tokens":1975,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":575,"completion_tokens_details":{"reasoning_tokens":1926}},"tokens_in":575,"tokens_out":1975,"duration_ms":14589,"temperature":1.0,"reasoning_tokens":1926,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:47:02.876118+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain only the PCE classifiers with negative samples generated by a different, independently trained generator (or by real videos with time-shifted audio), keep the rest of PAHA fixed, and rerun the evaluation. If the FGD, BAS, and Sync-C gains vanish or reverse, the guidance was exploiting artifacts of the fixed 60k-step generator rather than learning general audio-motion correspondence.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the primary co-speech gesture video dataset and the four-talker training/evaluation configuration used for the backbone and classifiers."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the pre-trained latent video diffusion model from which the unified backbone is initialized and whose 3D-UNet structure is adopted."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the state-of-the-art co-speech gesture video baseline and the training subset that PAHA reuses for fair comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the hand/face/body region decomposition that PAR's Awareness Areas are based on."}],"review_version":1}