{"id":"08d7f146-8243-4cfd-ad4d-cb81f1ad2fb7","arxiv_id":"2507.20987","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"A paper announcing a large-scale whole-body talking avatar benchmark and evaluation protocol, but with insufficient details to verify the dataset or the joint audio-video evaluation.","lead":"This paper introduces JWB-DH-V1, a claimed dataset of 2 million video clips of 10,000 identities for whole-body talking avatar research, along with evaluation protocols for video and speech quality. It reports that face and hand generation outperform whole-body generation, but provides little detail on the dataset or on joint audio-video assessment, making the central claims hard to verify.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The evaluation protocol is not joint: §3.2–3.4 define only video-only and audio-only metrics, and §4 excludes the sole joint model, so the benchmark does not assess joint audio-video generation as claimed.","rationale":"The reader's weakest assumption correctly identifies that the evaluation protocol does not measure joint generation. Section 3.2 and 3.3 define video-only metrics; Section 3.4 defines audio-only metrics; Section 4 excludes Veo-3, the only joint model. Thus the abstract's promise of assessing 'joint audio-video generation' is not realized. This is a correctness-level issue: the benchmark's stated purpose is falsified by its own evaluation design, independent of the dataset's existence. I considered whether the more load-bearing concern is that the 10,000-identity/2M-sample dataset is described in one sentence without curation details, but that concern is about evidence quality rather than the protocol's logical gap; the protocol issue is both concrete and sufficient to reject the paper's central claim. The verdict should remain REJECT, and because the reader already reached that verdict, no adjustment is needed. A revised paper could reframe the contribution as a unimodal video-quality benchmark plus a separate TTS benchmark, but as written the joint-generation claim is unsupported.","tokens_in":8354,"tokens_out":3563,"duration_ms":42382,"concrete_test":"Download the public repository and enumerate all evaluation scripts. If no script accepts both a generated video file and its corresponding audio waveform as inputs for a single score (for example, a lip-sync or audio-visual synchronization metric), the protocol is not joint. Additionally, run the current protocol on a fixed generated video with two different audios: if all reported scores are unchanged, then no metric in the protocol measures joint audio-video generation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that JWB-DH-V1 provides an 'evaluation protocol for assessing joint audio-video generation of whole-body animatable avatars.' That claim is not supported by the protocol actually defined. Section 3.2 and 3.3 specify twelve video metrics (SC, BC, MS, DD, AQ, IQ, FID, FVD, SSIM, PSNR, E-FID, CSIM), all of which take only generated video as input; Section 3.4 specifies audio-only TTS metrics (WER and LALM-based win-rate), which take only generated audio. No metric in the paper couples a generated video with its corresponding generated audio, neither as a joint score nor as a synchronization check. Moreover, Section 4 explicitly excludes Veo-3, the only evaluated joint audio-video model, 'due to instability in generating synchronized outputs from a single frame.' Consequently, Table 2 is purely visual and Table 1 purely audio; the protocol cannot detect audio-video synchronization failures, cross-modal consistency, or any property of joint generation. Even if the dataset exists at the stated scale, the benchmark's central evaluation contribution is absent as presented.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript introduces JWB-DH-V1, described as a large-scale benchmark for joint whole-body talking-avatar and speech generation, with a dataset of 10,000 identities, about 200 scene configurations per identity, and 2 million video clips, plus an evaluation protocol and an initial evaluation of several video-generation, talking-avatar, and TTS models. The paper reports separate video-quality metrics (SC, BC, MS, DD, AQ, IQ, FID, FVD, SSIM, PSNR, E-FID, CSIM) over whole-body, face, and hand regions (Table 2) and audio metrics WER and LALM-based win-rate (Table 1). Its stated conclusion is that current methods show consistent performance disparities between face/hand-centric and whole-body performance, indicating areas for future research. The dataset and evaluation tools are said to be publicly available at a GitHub repository.","tokens_in":8585,"tokens_out":4171,"duration_ms":42574,"significance":"Assessed as a benchmark paper, the contribution is currently a comparatively small empirical comparison rather than a validated resource. The authors deserve credit for making an associated GitHub repository available, for evaluating a broad set of recent video and TTS models, and for reporting results separately for whole-body, face, and hand regions, which is a useful format for diagnosing generation failures. If the dataset existed at the claimed scale and the protocol actually measured joint audio-video quality, JWB-DH-V1 would be a valuable addition. However, as described, the dataset curation is unverifiable and the evaluation protocol does not assess joint generation, so the main claims are not supported.","major_comments":[{"comment":"The central resource claim—'10,000 unique identities' and '2 million samples of video clips'—is presented without any description of data collection, annotation procedure, quality control, licensing, train/test split, or validation of the annotations shown in Figure 1. Because the paper's value as a benchmark depends on the dataset being curated and trustworthy, this omission makes the resource's existence and utility unassessable. A dataset card or a detailed curation section with annotation statistics is required.","section":"Section 3.1"},{"comment":"The evaluation protocol is not joint. All twelve video metrics listed in Sections 3.2-3.3 and all audio metrics in Section 3.4 take only one modality as input; no metric compares a generated video with its paired generated audio, and no synchronization, lip-sync, or cross-modal consistency measure is defined. Section 4 then excludes Veo-3, the only joint audio-video model, 'due to instability in generating synchronized outputs from a single frame.' Consequently, the evaluation cannot substantiate the Abstract's claim of 'an evaluation protocol for assessing joint audio-video generation of whole-body animatable avatars.'","section":"Sections 3.2-3.4 and Abstract"},{"comment":"Several entries are numerically implausible and suggest a preprocessing or scaling error. For instance, the whole-body row for Ha3/wo reports PSNR = 0.83 and SSIM = 0.016, with the hand region reporting PSNR = 0.49 and SSIM = 0.008, while other models in the same table report PSNR around 6-20 dB and SSIM 0.2-0.9. PSNR values below 1 dB are outside the normal range for images represented in [0,1] or [0,255], so the reported numbers cannot be trusted as meaningful quality scores. The conclusions in Section 4 rely on comparisons across these rows and must be recomputed.","section":"Table 2"},{"comment":"The protocol is not reproducible as written. The paper does not specify the number and selection of evaluation clips, video resolution and duration, generation prompts, random seeds, or the exact implementation and reference statistics for FID/FVD/E-FID. It also does not say how the region-specific masks (face, hand, whole body) are obtained or how many samples are used per model. These details are necessary for a benchmark paper.","section":"Section 4"}],"minor_comments":[{"comment":"There are several typos and formatting issues: 'incidates' should be 'indicates', 'with10,000' needs a space, and 'Version I(JWB-DH-V1)' needs proper formatting.","section":"Abstract"},{"comment":"The baseline row gpt-4o-mini-tts has a dash for win-rate, but the text in Section 3.4 says candidates are compared against a strong baseline; define how ties and 'winner=0' are handled in the win-rate formula.","section":"Table 1 and Section 3.4"},{"comment":"The statement '20,000 samples are used for evaluation' is ambiguous; it should be clarified whether this is the test split, a subset, or the total evaluation set.","section":"Section 3.1"},{"comment":"The reference list contains 'contentReference[oaicite:...]' artifacts (e.g., [1], [7], [25]), which should be removed before publication.","section":"References"}],"recommendation":"reject","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: the paper promises a 2M-clip, 10k-identity whole-body talking avatar benchmark and a joint audio-video evaluation protocol, but as written it delivers neither. The dataset is a single sentence; the protocol evaluates video and audio separately; and the one joint model is excluded. The stress-test concern is accurate.\n\nWhat is actually new: the region-specific evaluation idea — reporting per face, hand, and whole body — is sensible, and the paper runs eight models through twelve video and four audio metrics. If the dataset existed, the per-region comparison could be useful for locating where models fail.\n\nWhat the paper does well: the metric choices are standard and reproducible in principle; the TTS win-rate setup is standard; the writing is short and to the point.\n\nSoft spots, in order of severity.\n\nFirst, the load-bearing claim is unsubstantiated. Section 3.1 is one sentence: 10,000 identities, ~200 scene configurations, 2M clips, 20k for evaluation. No curation, no identity sampling, no annotation protocol, no samples, no dataset card, no code link with evidence. The GitHub URL is unverified. For a benchmark paper, this is fatal as presented.\n\nSecond, the evaluation is not joint. All twelve video metrics take generated video as input; the audio metrics take generated audio. No metric couples a video with its own audio, no synchronization check. Veo-3 is excluded, so the only joint model is never scored. The abstract's claim of a joint protocol is false.\n\nThird, the numbers are implausible. Table 2 has PSNR 0.83 and SSIM 0.016 for Ha3/wo whole body; that's not degraded video, that's broken evaluation. Several DD values near zero suggest the pipeline is not doing what the paper says.\n\nFourth, the paper offers no comparison to existing benchmarks or a clear task definition. Which model receives which inputs for the video metrics is only described loosely.\n\nBottom line: the paper should not be published as is. The region-specific idea is worth keeping, but the dataset, code, and a real joint metric are prerequisites. I'd desk reject this version and invite a resubmission with evidence.","headline":"Claims a large joint whole-body avatar benchmark but shows neither the dataset nor a joint evaluation; region-specific metrics are the one useful idea.","tokens_in":9096,"tokens_out":2883,"would_cite":false,"duration_ms":27901,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces JWB-DH-V1, a two-million-clip dataset and a three-track evaluation protocol meant to benchmark joint whole-body avatar and speech generation.","keywords":["whole-body avatar","talking avatar","speech generation","video generation","diffusion models","benchmark dataset","joint audio-video generation","evaluation protocol"],"falsifier":"Generate a clip in which the body and lip motion are frame-perfect but the audio track is shifted by a fixed delay; the benchmark's video and audio sub-scores would be unchanged while a human viewer would immediately see the synchronization is broken. If the protocol cannot flag this mismatch, it does not measure joint audio-video generation.","tokens_in":8142,"feed_emoji":"🎬","tokens_out":7774,"duration_ms":80810,"temperature":0.7,"pith_summary":"The authors are trying to establish that JWB-DH-V1 is the missing evaluation resource for whole-body talking avatars and speech generation, and that it is large enough and annotated richly enough to support region-specific analysis. If true, researchers would gain a common dataset and scoring protocol for comparing systems that must synthesize a full body plus matching speech, and the paper's initial results would show that current models are weakest precisely where whole-body realism matters. The paper's contribution is the resource and the protocol, not a new generative model.","feed_headline":"Two-million-clip benchmark targets whole-body talking avatars","feed_subtitle":"Evaluation of eight models shows face and hand quality still outpaces whole-body realism.","key_machinery":"The central object is the curated dataset itself, with its region-specific annotations, paired with a three-track evaluation protocol: reference-free video metrics, co-speech fidelity metrics, and a Large-Audio-Language-Model win-rate for speech. The dataset is what enables the region-specific breakdown; the protocol is what turns raw clips and audio into comparable numerical scores.","core_discovery":"The central claim is that the field lacks a benchmark tailored to joint whole-body avatar and speech generation, and that JWB-DH-V1 provides one. The dataset is described as containing 10,000 unique identities, each in roughly 200 scene configurations, for about 2 million video samples with annotations including segmentation, landmarks, bounding boxes for hands and legs, motion text, speech transcription, word boundaries, and ground truth audio; 20,000 samples are held out for evaluation. The protocol has three tracks: reference-free video metrics, co-speech fidelity metrics (FID, FVD, SSIM, PSNR, E-FID, CSIM), and speech audio evaluation using WER plus a Large-Audio-Language-Model win-rate. On this protocol, eight generative models are scored across face, hand, and whole-body regions, and the results show a consistent gap between face/hand quality and whole-body quality. The only joint audio-video model considered, Veo-3, is excluded because its outputs from a single frame were unstable.","pith_inferences":["The exclusion of the only joint model implies that current joint generation is too unstable to benchmark; a version that includes stable joint models would be needed to test the joint claim.","Because the protocol scores video and audio separately, a model with perfect synchronization but mediocre per-track scores would not be recognized; adding a synchronization-aware metric or human study is a natural extension.","The region annotations could let researchers ask whether models over-fit to faces at the expense of hands and whole-body, or whether whole-body errors stem from data imbalance."],"forward_implications":["Region-specific scoring shows consistent performance gaps between face/hand and whole-body regions, pointing to where generative models need work.","The benchmark's 2 million annotated clips and 20,000 evaluation samples provide a common ground for comparing whole-body avatar generation systems.","The video and speech protocols yield numerical scores that can be reported and compared without human ratings.","Future versions that extend clips to 60 seconds would test long-horizon synchronization rather than short clips."],"supporting_citations":[{"why":"It is the only true joint audio-video generation model considered, and its exclusion due to instability defines the gap the benchmark is meant to fill.","marker":"[6]"},{"why":"It supplies the Large-Audio-Language-Model model-as-a-judge framework used for the speech audio evaluation.","marker":"[22]"},{"why":"It is the source of the FID, E-FID, and CSIM metrics used in the co-speech sub-protocol.","marker":"[28]"},{"why":"It defines Fréchet Video Distance, the temporal fidelity metric used in the co-speech sub-protocol.","marker":"[36]"},{"why":"It provides DINO features that the Subject Consistency score is computed from.","marker":"[3]"},{"why":"It provides CLIP features that the Background Consistency score is computed from.","marker":"[30]"},{"why":"It is the optical flow method used for the Motion Smoothness score.","marker":"[20]"},{"why":"It is the LAION aesthetic predictor used for the Aesthetic Quality score.","marker":"[17]"},{"why":"It is the MUSIQ model used for the Imaging Quality score.","marker":"[15]"}],"fun_headline_variants":["2M-clip benchmark for whole-body avatars and speech","New benchmark exposes whole-body avatar gap","Benchmark with 2M clips tests joint avatar-speech","Whole-body avatar benchmark highlights face-hand gap","JWB-DH-V1: 2M videos for avatar-speech evaluation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark assumes that a protocol which scores video and audio in separate tracks, and which excludes the only joint generation model, still measures joint audio-video generation quality.","fun_headline_variants_meta":{"raw":{"variants":["2M-clip benchmark for whole-body avatars and speech","New benchmark exposes whole-body avatar gap","Benchmark with 2M clips tests joint avatar-speech","Whole-body avatar benchmark highlights face-hand gap","JWB-DH-V1: 2M videos for avatar-speech evaluation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000503,"raw_usage":{"total_tokens":2439,"prompt_tokens":906,"completion_tokens":1533,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":522,"completion_tokens_details":{"reasoning_tokens":1452}},"tokens_in":522,"tokens_out":1533,"duration_ms":12628,"temperature":1.0,"reasoning_tokens":1452,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T13:03:35.185963+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Generate a clip in which the body and lip motion are frame-perfect but the audio track is shifted by a fixed delay; the benchmark's video and audio sub-scores would be unchanged while a human viewer would immediately see the synchronization is broken. If the protocol cannot flag this mismatch, it does not measure joint audio-video generation.","supporting_citations":[{"cited_title":"DeepMind","cited_arxiv_id":null,"evidence_quote":"It is the only true joint audio-video generation model considered, and its exclusion due to instability defines the gap the benchmark is meant to fill."},{"cited_title":"Caron, H","cited_arxiv_id":null,"evidence_quote":"It provides DINO features that the Subject Consistency score is computed from."},{"cited_title":"Radford, J","cited_arxiv_id":null,"evidence_quote":"It provides CLIP features that the Background Consistency score is computed from."},{"cited_title":"Li, Z.-L","cited_arxiv_id":null,"evidence_quote":"It is the optical flow method used for the Motion Smoothness score."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It is the MUSIQ model used for the Imaging Quality score."}],"review_version":1}