{"id":"5eaa58af-c7ff-4b8e-af1a-f14618763454","arxiv_id":"2507.23343","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A new large dataset and the FSCD model improve automated quality scoring of AI-generated talking-head videos, beating 15 baselines in correlation with human ratings.","lead":"This paper introduces THQA-10K, a dataset of 10,457 AI-generated talking-head videos scored by 25 human raters, and proposes FSCD, a model that predicts quality from the first frame, a Y-T slice, and lip-sync features. FSCD reports higher correlation with human opinion than 15 existing quality metrics on three talking-head datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Y-T slice mouth alignment via Eq. (5) is unverified for cropped/warped talkers; a per-talker landmark check could change the SOTA margin.","rationale":"The reader identified the Y-T slice alignment as the weakest assumption. On close reading, this is indeed the most load-bearing unverified aspect of the central SOTA claim. The method's contribution is both a dataset and a metric; the metric's advantage depends on the slice cutting through the mouth. Eq. (5) assumes talkers preserve portrait geometry up to uniform scale, but talker outputs vary in resolution, cropping, and spatial layout (Table 3). The paper reports no alignment check, and the ablation quantifies the slice's contribution (~0.07 SRCC on THQA-10K), so the assumption is not decorative. The concern is not about integrity but about a specific geometric precondition that is unverified. The proposed test, per-talker landmark-based verification, settles whether the precondition holds. If misalignment is found, FSCD may need direct landmark detection on video rather than scaling from the portrait. The dataset construction and experiments are otherwise sound, so conditional acceptance remains appropriate. The verdict is CONDITIONAL, matching the reader's, because the concern is concrete and addressable with a focused verification study.","tokens_in":23831,"tokens_out":1702,"duration_ms":18702,"concrete_test":"Sample ~200 AGTHs across all 14 talkers in THQA-10K. Run a facial landmark detector directly on video frames (first frame and a few later frames), compute the true mouth centroid Cv_true, and compare to Cv from Eq. (5). Report per-talker median Euclidean distance normalized by face width. If median distance exceeds ~5% of face width for any talker, recompute FSCD performance using Cv_true and check whether SRCC changes materially; this would show whether the unverified alignment assumption affects the SOTA claim.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The SOTA claim rests on FSCD beating 15 baselines on three datasets, with the Y-T slice as a key temporal feature. The slice position is computed in Eq. (5) by scaling the T2I portrait mouth centroid Co by Rv/Ro. This assumes the talker output is a simple uniform scaling of the portrait, preserving relative mouth location. However, talkers crop, pad, warp, translate, or change aspect ratio (Table 3: resolutions range 256x256 to 1024x1024; Wav2Lip and Video-Retalking use 1024 outputs from 512 or 1024 inputs, often with letterboxing; StyleHeat uses StyleGAN and may not preserve the portrait framing). If the centroid is misplaced, the Y-T slice cuts through cheek or background, degrading the temporal feature. The paper never verifies per-talker alignment. Table 5 shows removing the slice (w/oS) drops THQA-10K SRCC from 0.8066 to 0.7330, so the slice contributes ~0.07 SRCC. If the slice is misaligned for many videos, the reported number understates what a properly aligned slice could achieve; the comparison with baselines remains fair, but the method's mechanism is misattributed. The reader's concern is valid: Eq. (5) is load-bearing and unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents THQA-10K, a dataset of 10,457 AI-generated talking-head videos produced by 12 text-to-image models and 14 speech-driven talkers, with subjective MOS and distortion labels collected from 25 raters. The authors analyze how talker generalization and output quality vary, and propose FSCD, an objective quality regression model that concatenates the first video frame, a Y-T temporal slice, and SyncNet-derived LSE-C/LSE-D maps into a single image and regresses it to MOS with a Swin-T backbone. On THQA-10K, THQA, and THQA-3D, FSCD is reported to achieve SRCC of 0.8066, 0.7812, and 0.8235, outperforming 15 baselines, and ablations show each of the four components contributes.","tokens_in":24073,"tokens_out":7974,"duration_ms":51958,"significance":"The dataset is a substantial contribution: it is the largest AGTH quality assessment dataset to date, covers a wide range of modern generative models, and its subjective data enable analysis of talker generalization and distortion prevalence. The release of the dataset and code is a positive step for the community. FSCD's design is novel in treating the Y-T slice as an image-level temporal feature and in feeding SyncNet scores through the same backbone as the spatial frame. If the statistical and methodological concerns below are addressed, the paper would be a useful benchmark for AGTH quality assessment.","major_comments":[{"comment":"The Y-T slice position is computed by scaling the mouth centroid Co of the T2I portrait by the resolution ratio Rv/Ro (Eq. 5), which assumes that the talker output is a uniformly scaled version of the portrait with no translation, cropping, letterboxing, or aspect-ratio change. The talkers in Table 3 include methods that output 1024×1024 from 512×512 inputs (Wav2Lip, Video-Retalking, DINet, IP-LAP, TalkLip, MuseTalk) and StyleGAN-based methods (StyleHeat), for which such preprocessing is common. The paper never verifies that Cv falls inside the mouth region for each AGTH. Since Table 5 shows that removing the Y-T slice reduces THQA-10K SRCC from 0.8066 to 0.7330, the slice is a major contributor to the reported performance; an unverified alignment assumption is therefore load-bearing. Please validate the alignment, for example by running a landmark detector on AGTH frames and reporting the distance between Cv and the true mouth centroid per talker, and if misalignment is found, correct the slice placement or exclude misaligned cases.","section":"Section 4.1, Eq. (5)"},{"comment":"The SOTA claim is based on SRCC margins of +0.029 (THQA-10K), +0.021 (THQA), and +0.036 (THQA-3D) over the second-best methods, but only the averaged metrics over five folds are reported, without standard deviations or significance tests. The five-fold average can be sensitive to a single fold, and the margin is within typical cross-fold variability. Please report per-fold results with mean±std for all four metrics and run a paired significance test (e.g., Wilcoxon signed-rank) between FSCD and the top baselines on each dataset. Without this, the claim that FSCD surpasses the next best algorithm by at least +2% SRCC is not statistically grounded.","section":"Section 5.1, Table 4"},{"comment":"FSCD uses the T2I source portrait to compute the mouth centroid Co, and therefore requires access to the original portrait at test time. The 15 baselines in Table 4 are no-reference methods (with LSE-C/LSE-D using only the video and audio). This makes the comparison asymmetric: FSCD is effectively a reduced-reference method, since the source image provides information about the intended framing and identity. The paper should either (a) compute the mouth centroid from the AGTH video itself, for example by applying a face landmark detector to the first frame of the AGTH, which would make FSCD no-reference; or (b) explicitly frame FSCD as reduced-reference and compare with appropriate baselines. If option (a) is taken, the experiments and ablations need to be rerun.","section":"Section 4.1, Eqs. (4)-(5)"},{"comment":"The paper states that the five-fold data partitioning ensures that there is no content overlap, but it does not specify the unit of partitioning. If splits are video-level random, the same portrait and audio can appear in both training and test sets (for different talkers), which would constitute content leakage because FSCD's first-frame and slice features are strongly tied to the portrait. Please specify whether the split is at the level of prompt (PID), portrait, or audio, and demonstrate that no content from one fold appears in another. If the current splits are video-level, please re-split at the portrait or prompt level and rerun the experiments.","section":"Section 5.1, data partitioning"}],"minor_comments":[{"comment":"The text says 'identify the key points pk of the mouth' but the variable description says 'K denotes the total number of key points on the face.' Please correct to 'on the mouth.'","section":"Section 4.1, Eq. (4)"},{"comment":"The arrangement of the four FSCD input components (first frame, Y-T slice, LSE-C, LSE-D) is not specified; the paper should state whether they are tiled as a 2×2 grid, concatenated along the channel dimension, or arranged otherwise, and give the resizing and normalization steps.","section":"Section 4.3, Fig. 8"},{"comment":"The table lists 'Head Motion' but the paper does not explain how head-motion presence affects the Y-T slice extraction or quality; a sentence clarifying whether the slice is computed over the whole frame regardless of head motion would be helpful.","section":"Section 3.2, Table 3"},{"comment":"The conclusions state that the next best algorithm is surpassed by at least +2% SRCC, but the second-best method differs per dataset; the text should name the second-best method per dataset for clarity.","section":"Section 5.2"},{"comment":"The phrase 'figuratively known as Talkers' is informal; consider rephrasing for a more formal tone.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The paper's main claims are contingent on resolving the reference-setting and split issues; if those are resolved, the dataset alone is a strong contribution. The authors are also authors of THQA [72] and THQA-3D [73], so the comparison is with their own prior benchmarks; this is not a problem, but the novelty relative to those works should be stated more clearly in the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about arXiv:2507.23343. First, the THQA-10K dataset is the real contribution: 10,457 AI-generated talking-head videos from 12 T2I models and 14 talkers, with MOS and distortion labels from 25 subjects. That is a solid, reusable asset for anyone working on digital human quality assessment. Second, the proposed FSCD metric is a sensible combination of established pieces—first frame, Y-T slice, SyncNet LSE-C/LSE-D maps—stacked into a single image for a Swin-T regressor. It reports state-of-the-art SRCC on three datasets (0.8066 on THQA-10K, 0.7812 on THQA, 0.8235 on THQA-3D), with ablations showing each component contributes.\n\nThe paper is careful in several ways: the subjective experiment follows ITU-R BT.500, z-score processing is standard, five-fold cross-validation ensures no content overlap, and the code and data are promised at a public repo. The feature choices are motivated by distortions observed in the same dataset, but because FSCD is a supervised regressor evaluated on held-out folds, that is not circular; it is standard practice.\n\nThe main soft spot is the Y-T slice alignment. Equation (5) places the slice through the mouth by linearly scaling the T2I portrait's mouth centroid by the resolution ratio Rv/Ro. That assumes the talker output is a uniform scaling of the portrait. Table 3 shows output resolutions from 256×256 to 1024×1024, and several talkers (Wav2Lip, Video-Retalking, StyleHeat) crop, pad, warp, or change framing. If the centroid is misplaced, the slice cuts through cheek or background, degrading the temporal feature. The paper never verifies per-talker alignment on actual AGTH frames. The ablation shows removing the slice costs about 0.07 SRCC on THQA-10K, so the slice matters; but if it is misaligned for many videos, the reported number may understate what a properly aligned slice could do. The comparison with baselines remains fair, but the mechanism as described is unverified.\n\nA secondary, minor issue: no standard deviations or significance tests across folds. The margins over the next best baseline are around 2–4% SRCC, which could be meaningful, but without error bars we cannot tell.\n\nWho is this for? Researchers building or evaluating talking-head generation systems, and the QA community. The dataset alone justifies referee time. FSCD is a reasonable baseline and likely to be cited. Recommendation: send it to peer review. The authors should be asked to verify slice alignment per talker with landmarks on actual frames and report fold-level statistics before the SOTA claim is taken at face value.","headline":"Large AGTH quality dataset is the real contribution; FSCD is a solid metric but its Y-T slice alignment assumption is unverified and should be checked before the SOTA claim is accepted.","tokens_in":24661,"tokens_out":3902,"would_cite":true,"duration_ms":35657,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single composite image made of a first frame, a vertical time slice through the mouth, and two lip-sync error maps can predict human quality ratings for AI-generated talking heads and beats 15 comparison methods on three datasets.","keywords":["AI-generated talking heads","talking head quality assessment","THQA-10K dataset","Y-T slice","lip-sync consistency","SyncNet","Swin Transformer","mean opinion score"],"falsifier":"Measure the true mouth position in the videos of THQA-10K and compare it with the scaled portrait centroid used to place each Y-T slice; if many slices miss the mouth while FSCD still achieves its reported 0.81 SRCC, the slice is not doing the claimed temporal work, whereas if re-centering slices on true mouth landmarks raises accuracy, the alignment assumption is load-bearing.","tokens_in":23617,"feed_emoji":"🗣️","tokens_out":3730,"duration_ms":43988,"temperature":0.7,"pith_summary":"The paper sets out to answer who generates better AI talking heads by building the largest dataset of such videos to date and an automatic way to score them. It claims that one composite image, formed by stacking a video's first frame, a vertical time slice through the mouth, and two lip-sync error maps, carries enough information to predict how human viewers will rate the video. If that claim holds, talker development can be guided by an objective score instead of slow subjective testing, and quality differences between speech-driven methods can be measured reliably. The paper supports the claim with state-of-the-art rank correlations of 0.8066, 0.7812, and 0.8235 on three talking-head quality datasets.","feed_headline":"One composite image predicts AI talking-head quality","feed_subtitle":"FSCD stacks first frame, a mouth time slice, and lip-sync maps, topping 15 methods on three datasets.","key_machinery":"The load-bearing object is the FSCD composite image, assembled from four aligned inputs: the first frame, a Y-T slice, and two SyncNet consistency maps. The Y-T slice is a vertical column of pixels taken at the mouth's horizontal position and laid out over time, so mouth motion becomes a static texture that a spatial backbone can read. The mouth position is found by detecting landmarks in the T2I-generated still portrait and scaling that centroid by the video-to-image resolution ratio (Eq. 5), and SyncNet's LSE-C and LSE-D expand into two additional image channels instead of being regressed with fixed weights. Swin-T then extracts quality features from the composite and predicts the MOS directly.","core_discovery":"The central discovery is FSCD, an objective quality assessment method that treats an AI-generated talking head as a single composite image rather than as a sequence of frames. The composite stacks the first frame (spatial quality), a Y-T slice cut through the mouth column over time (temporal mouth motion), and the LSE-C and LSE-D outputs of SyncNet (tone-lip consistency), and feeds all four channels into a Swin-T regression network trained with mean squared error against human mean opinion scores. The paper reports that FSCD outperforms all fifteen compared image, video, and lip-sync quality metrics on the new THQA-10K dataset and on the earlier THQA and THQA-3D benchmarks, and its ablations show that each of the four input components contributes positively to performance.","pith_inferences":["An implicit extension is that the same composite trick could compress any temporally structured generative video into a single image for quality prediction, which may benefit other domains such as lip-synced dubbing or expressive avatars without requiring video backbones.","Because the first frame is the most important component and many talkers keep identity static, FSCD may work best for near-static talking heads and could lose accuracy on methods with large head motion or camera motion, where a single vertical slice no longer tracks the mouth.","The paper's own distortion analysis shows misaligned keypoints have the largest perceptual impact, so a direct misalignment detector, trained on landmark displacement rather than on MOS, could complement FSCD and give talker developers an actionable fault signal.","A testable consequence not reported in the paper is that re-centering the Y-T slice on mouth landmarks detected directly in each generated video, rather than scaling the portrait centroid, could further improve accuracy for talkers that crop or translate the face."],"forward_implications":["If FSCD is right, talker design can be iterated against an automatic score instead of repeated human rating sessions, since the composite image predicts perceived quality at state-of-the-art accuracy.","Different talkers differ substantially in output quality, with EchoMimic and SadTalker rated highest and EmoGen and StyleHeat lowest, while choice of T2I model matters less, so talker selection can be based on the FSCD score.","The 10,457-video dataset records 17,191 distortions, with blur, noise, and artifacts most common but misaligned keypoints most damaging to perceived quality, giving talker developers a concrete list of failure modes to fix.","Even without the first frame, the Y-T slice plus lip-sync features still achieve competitive accuracy, so temporal mouth-motion information alone is sufficient for a useful quality signal.","FSCD generalizes across three datasets with different content and distortion mixes, suggesting the composite-image format transfers beyond the specific T2I models and talkers used to build THQA-10K."],"supporting_citations":[{"why":"Supplies the prior THQA dataset with 800 AGTHs and nine distortion types that THQA-10K extends and that FSCD is evaluated against.","marker":"[72]"},{"why":"Introduces the X-T/Y-T slice representation for temporal features that FSCD adapts into a mouth-centered Y-T slice.","marker":"[38]"},{"why":"Provides SyncNet and the LSE-C and LSE-D lip-sync consistency features that FSCD expands into two input image channels.","marker":"[8]"},{"why":"Defines the Swin-T transformer backbone used to regress quality from the FSCD composite image.","marker":"[22]"},{"why":"Provides the THQA-3D dataset used as a third benchmark for evaluating FSCD's generalization.","marker":"[73]"},{"why":"Sets the ITU-R BT.500 protocol that governs the subjective rating collection and subject rejection in the THQA-10K experiment.","marker":"[4]"}],"fun_headline_variants":["Composite image beats 15 metrics for talking-head quality","FSCD: one fused image scores AI talking-head quality","Single image stack predicts talking-head quality best","Fusing frame, slice, lip-sync yields SOTA talking-head score"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that scaling the mouth center found in the still portrait by the video-to-image resolution ratio puts the vertical slice through the mouth in every generated video, even when a talker crops, warps, translates, or changes the aspect ratio.","fun_headline_variants_meta":{"raw":{"variants":["Composite image beats 15 metrics for talking-head quality","FSCD: one fused image scores AI talking-head quality","Single image stack predicts talking-head quality best","Fusing frame, slice, lip-sync yields SOTA talking-head score"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000175,"raw_usage":{"total_tokens":1298,"prompt_tokens":968,"completion_tokens":330,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":584,"completion_tokens_details":{"reasoning_tokens":262}},"tokens_in":584,"tokens_out":330,"duration_ms":4474,"temperature":1.0,"reasoning_tokens":262,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T10:49:32.763633+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the true mouth position in the videos of THQA-10K and compare it with the scaled portrait centroid used to place each Y-T slice; if many slices miss the mouth while FSCD still achieves its reported 0.81 SRCC, the slice is not doing the claimed temporal work, whereas if re-centering slices on true mouth landmarks raises accuracy, the alignment assumption is load-bearing.","supporting_citations":[{"cited_title":"Thqa: A perceptual quality assessment database for talking heads","cited_arxiv_id":null,"evidence_quote":"Supplies the prior THQA dataset with 800 AGTHs and nine distortion types that THQA-10K extends and that FSCD is evaluated against."},{"cited_title":"An xt slice based method for action recognition","cited_arxiv_id":null,"evidence_quote":"Introduces the X-T/Y-T slice representation for temporal features that FSCD adapts into a mouth-centered Y-T slice."},{"cited_title":"Out of time: auto- mated lip sync in the wild","cited_arxiv_id":null,"evidence_quote":"Provides SyncNet and the LSE-C and LSE-D lip-sync consistency features that FSCD expands into two input image channels."},{"cited_title":"Swin transformer: Hierarchical vision transformer using shifted windows","cited_arxiv_id":null,"evidence_quote":"Defines the Swin-T transformer backbone used to regress quality from the FSCD composite image."},{"cited_title":"Subjective and objective quality-of-experience assessment for 3d talking heads","cited_arxiv_id":null,"evidence_quote":"Provides the THQA-3D dataset used as a third benchmark for evaluating FSCD's generalization."},{"cited_title":"Methodology for the subjective assessment of the quality of television pictures","cited_arxiv_id":null,"evidence_quote":"Sets the ITU-R BT.500 protocol that governs the subjective rating collection and subject rejection in the THQA-10K experiment."}],"review_version":1}