{"id":"c8b6e43f-8826-4a77-81c4-3c3b6e86c3ec","arxiv_id":"2507.06060","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"VisualSpeaker supervises photorealistic 3D Gaussian Splatting renders with a pretrained lip-reading model, reducing lip vertex error by 56.1% on MEAD versus an ablated baseline and improving perceived lip clarity in a user study.","lead":"This paper introduces VisualSpeaker, a method that trains 3D avatars to move their lips by checking photorealistic renders with a lip-reading AI. The approach cuts lip vertex error by 56.1% on MEAD compared to the authors' own baseline, and the authors say it can help sign language avatars produce readable mouthings.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported 56.1% LVE improvement is confounded by extra training: the 'w/o Lread' baseline is the stage-2 model, while the full model receives 100 additional epochs on MEAD; no Lvert-only control isolates the loss's effect.","rationale":"I chose the training-time confound as the most load-bearing concern because it directly infects the paper's headline number. The strongest claim is that applying Lread improves both LVE and perceived quality. Table 1's 56.1% improvement compares the stage-2 model (w/o Lread) with the stage-3 model (with Lread). Since stage 3 adds 100 epochs of MEAD training, the comparison conflates the loss with additional optimization. Without an Lvert-only control for the same extra epochs, the causal role of Lread is not established. This is not an attack on the authors' integrity; it is a standard experimental-design issue that a single retraining experiment can resolve. If the control shows little LVE change, the paper's central claim is supported; if not, the claimed effect shrinks. The reader's weakest assumption about the AutoAVSR domain gap is also relevant, but it attacks the mechanism rather than the measurement: even if the embedding is misaligned on predicted renders, the loss could still accidentally improve LVE via extra training. The confound is thus more decisive for the specific quantitative claim. I still see value in the method and in the idea of pixel-space lip-reading supervision with 3DGS, so I would not reject outright; I recommend CONDITIONAL acceptance contingent on the Lvert-only control (and ideally a silent, visual-only lip-reading evaluation).","tokens_in":12722,"tokens_out":9729,"duration_ms":100491,"concrete_test":"Retrain the stage-2 model for exactly 100 additional epochs on MEAD using only Lvert (same effective batch size, learning rate, and vertex weighting as stage 3), then compute LVE on the MEAD test set and run the A/B user study against the stage-2 baseline. If the Lvert-only control reaches an LVE close to the 1.69 mm reported for the full model, or is preferred at a comparable rate, the 56.1% improvement is not specifically due to Lread; if it remains near 3.85 mm, the confound is minor.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 3.4 describes a three-stage curriculum: (1) VOCASET pretraining with Lvert, (2) MEAD fine-tuning with Lvert, (3) a further 100 epochs on MEAD with Lvert + λread·Lread. Table 1 compares the stage-2 model ('VisualSpeaker w/o Lread', LVE 3.85 mm) against the stage-3 model ('VisualSpeaker', 1.69 mm). The two models therefore differ not only in the presence of Lread but also in the amount of MEAD training: the full model receives roughly 100 additional epochs (batch-size-1 with 4-step accumulation, i.e. the same effective batch size as earlier stages). Any continued Lvert-only training could itself reduce LVE via better optimization, more data exposure, or reduced over-smoothing. The paper does not report a control trained for the same number of extra epochs with Lvert alone. Consequently, the 56.1% improvement cannot be unambiguously attributed to the perceptual loss. Because LVE is the paper's headline quantitative claim, this confound directly undermines the central assertion that the lip-reading loss improves geometric accuracy. The user study also compares the stage-3 model to the stage-2 baseline, inheriting the same confound; the audio-present protocol additionally makes 'lip clarity' judgments dependent on audio-visual sync rather than visual intelligibility alone.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes VisualSpeaker, an encoder-decoder transformer that predicts FLAME vertex offsets from audio or text and renders them through a per-subject 3D Gaussian Splatting avatar. During a third training stage, a perceptual lip-reading loss (Eq. 5) is computed by passing 96x96 lip crops of the rendered frames through a pretrained AutoAVSR model and comparing its embeddings with those of ground-truth video. The paper reports a 56.1% reduction in Lip Vertex Error on MEAD (Table 1), modest improvements in image metrics (Table 2), and a user study favoring the full model over the no-Lread baseline (Table 3). A text-to-mouthing application for BSL sign pairs is also demonstrated.","tokens_in":12977,"tokens_out":5072,"duration_ms":55447,"significance":"The core idea of supervising mesh-driven 3DGS avatars with a pretrained visual speech recognition model on rendered pixels is timely and potentially useful, especially for sign-language mouthing where lip readability is linguistically important. The paper is clearly written, the architecture is sensible, and the authors make a good-faith effort to validate the synthetic-to-real domain gap using an external embedding model and to support their claims with a user study. However, the central quantitative claim is currently undermined by a training-budget confound: the full model receives 100 additional epochs of MEAD training compared with the no-Lread baseline, and no Lvert-only control is reported. In addition, the paper compares only its own pipeline stages and not any published method. If the authors add the missing control and external comparisons, the contribution could become a solid systems paper; as it stands, the evidence does not yet establish that the lip-reading loss, rather than extra training, drives the reported improvements.","major_comments":[{"comment":"The headline 56.1% LVE improvement is confounded by unequal training budgets. The baseline 'VisualSpeaker w/o Lread' is the stage-2 model trained for 250 epochs on MEAD with Lvert only, while the full model is trained for an additional 100 epochs with Lvert + lambda_read * Lread (batch size 1, gradient accumulation over 4). A control trained with Lvert alone for the same additional 100 epochs and identical schedule is not reported. Without this control, the LVE reduction from 3.85 mm to 1.69 mm cannot be unambiguously attributed to Lread; it may simply reflect continued optimization on the MEAD training set. This is load-bearing for the abstract's central claim, so the authors should add an Lvert-only fine-tuning control and report its LVE.","section":"Section 3.4, Table 1"},{"comment":"The paper compares only its own successive pipeline stages and no published method, yet the introduction and conclusion claim that VisualSpeaker surpasses mesh-based baselines. No LVE numbers are reported for FaceFormer, CodeTalker, EMOTE, GaussianTalker, or other prior work on the same MEAD test split and pseudo-ground-truth fitting. Because LVE is a standard metric in this area, adding comparisons to at least one or two representative methods under the same evaluation protocol is necessary to support the claimed geometric improvement over the state of the art.","section":"Section 4.2, Table 1"},{"comment":"The domain-gap validation is performed only on renders of ground-truth meshes: the authors report cosine similarity 0.697 for matching pairs and 0.190 for mismatched pairs. During stage 3, however, Lread is applied to renders of predicted meshes, which are of lower quality early in training. The paper does not measure whether AutoAVSR embeddings remain aligned for such predicted-mesh renders, so it is possible that Lread rewards spurious visual features rather than lip articulation during the early part of the final stage. The authors should compute the same cosine-similarity statistic on predicted-mesh renders at representative points during training, or otherwise demonstrate that the loss gradients are not chasing out-of-distribution rendering artifacts.","section":"Section 3.2"},{"comment":"The user study inherits the same training-budget confound, because 'Ours vs. Baseline' compares the stage-3 model against the stage-2 model. Moreover, the study instructions state that 'Audio is provided with the videos,' so the 'lip clarity' ratings may reflect audio-visual synchronization and overall intelligibility rather than pure visual lip-readability. To support the claim that the perceptual loss improves lip readability, the authors should run a silent (audio-free) condition or at least report results broken down by audio-present vs audio-absent presentation.","section":"Section 4.3 and Supplementary 6.2"}],"minor_comments":[{"comment":"The symbol V is used for both the total vertex count and the vertex index; please rename one of them for clarity.","section":"Section 3.3, Eq. (4)"},{"comment":"The sentence 'is an key metric' contains a typo and should read 'is a key metric.'","section":"Section 4.2"},{"comment":"The phrase 'the retrained models performance on VOCASET' should be 'the pretrained model's performance on VOCASET.'","section":"Section 4.2"},{"comment":"The word 'ingrate' should be 'integrate' in the final sentence of Section 4.3.","section":"Section 4.3"},{"comment":"The description of how categorical preferences are converted to a {−2,...,2} score and then to 'percentage of times preferred' is underspecified; please clarify how ties and unequal numbers of raters per video are handled.","section":"Supplementary 6.2"}],"recommendation":"major_revision","confidential_remarks":"This is a promising systems paper, but the central quantitative claim is currently confounded by unequal training budgets and a lack of external baselines. I do not see evidence of misconduct; the ablation is simply incomplete. If the authors add an Lvert-only control, compare against at least one or two published methods, and clarify the user-study protocol, the paper could become a solid contribution. The scope fits a computer vision or graphics venue, and the sign-language application is a meaningful extension."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the core idea—using a lip-reading expert to supervise photorealistic renders of a 3D avatar—is sensible and worth pursuing, but the headline 56.1% LVE improvement cannot be attributed to the loss as presented. The comparison is between the stage-2 model (Lvert only, MEAD fine-tuned) and the stage-3 model (Lvert + Lread, 100 additional MEAD epochs). Without a control that trains Lvert alone for the same extra epochs, the gain could just be more training. That confound runs through the whole paper, including the user study, because the same two models are compared there.\n\nWhat's genuinely new: applying the lip-reading loss to photorealistic 3DGS renders, rather than mesh renders, is a real step beyond prior work. The paper is honest about related work, and the domain-gap check (cosine similarity 0.697 vs 0.190) is a nice sanity check. The text-to-mouthing demo for BSL is a good practical motivation and a plausible use case.\n\nWhere it's soft: no comparison to existing methods (FaceFormer, CodeTalker, GaussianTalker), so even if the confound were fixed, we'd only know the loss helps their pipeline, not where it stands relative to SOTA. No error bars on LVE, and the pseudo-GT is noisy. The user study tells participants audio is provided, so 'lip clarity' is hard to separate from audio-visual sync, and there's no significance testing. Also, the domain-gap validation only covers renders of ground-truth meshes; during training the predicted meshes are worse, and we don't know if the AVSR embeddings stay aligned. Circularity is low, since AutoAVSR is external, which is a point in their favor.\n\nThat said, the paper is clearly written, the curriculum is sensible, and the authors openly discuss limitations. The central claim is currently overstated, but the underlying idea is plausible and testable. A proper control experiment (same architecture, same total epochs, with and without Lread) and at least one external baseline would make this a solid paper.\n\nWho for: anyone working on 3D talking heads, perceptual losses, or sign-language avatar generation. It deserves a serious referee, but the referee should insist on the control and baselines before accepting the headline claim.","headline":"Sensible idea, but the headline 56.1% LVE improvement is confounded by extra training; the paper needs a proper control and baseline comparisons before that claim can be taken seriously.","tokens_in":13577,"tokens_out":3008,"would_cite":false,"duration_ms":30492,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"VisualSpeaker claims that supervising a 3D avatar's lip motions with a pre-trained lip-reading model, applied to photorealistic renders, cuts Lip Vertex Error by 56.1% on MEAD and improves perceived lip clarity.","keywords":["3D facial animation","lip-reading loss","perceptual supervision","3D Gaussian Splatting","visual speech recognition","FLAME model","sign language mouthing","lip vertex error"],"falsifier":"Measure the AutoAVSR cosine similarity between renders of the model's predicted meshes at an early epoch and their matching ground-truth frames. If the mean similarity of these matching pairs falls toward the mismatched-pair baseline of 0.190, the perceptual loss is not tracking lip articulation and the LVE improvement would not reflect true readability. A complementary test is a silent video lip-reading study pairing VisualSpeaker against the baseline at matched LVE; equal word accuracy would show the loss improves geometry but not perception.","tokens_in":12483,"feed_emoji":"👄","tokens_out":6664,"duration_ms":64763,"temperature":0.7,"pith_summary":"This paper proposes that a 3D talking avatar's mouth can be trained to be more accurate and more readable by having a pre-trained visual speech recogniser watch the rendered lips. The authors build VisualSpeaker, which predicts FLAME mesh deformations from audio or text, renders the mouth region as a photorealistic 3D Gaussian Splatting image, and adds a loss that pulls the recogniser's features for the render toward those of the ground-truth video. On the MEAD dataset, adding this lip-reading loss to a standard vertex loss lowers the Lip Vertex Error from 3.85 mm to 1.69 mm, a 56.1% reduction, and a user study prefers the resulting animations 65% of the time for realism and 66.6% for lip clarity. The motivation is that geometric losses alone reward averaged mouth shapes, whereas perceptual readability matters in applications like sign language, where mouthings disambiguate manual signs.","feed_headline":"Lip-reading AI grades avatar mouths, cutting lip error 56%","feed_subtitle":"A pre-trained visual speech recognizer supervises photorealistic renders, improving lip readability for signed and spoken language avatars.","key_machinery":"The load-bearing object is the perceptual lip-reading loss computed on photorealistic renders. The pipeline first deforms a FLAME mesh with tooth geometry via an autoregressive transformer decoder, conditioned on Wav2Vec2.0 audio features or F5-TTS text-to-speech output. The predicted mesh drives a 3D Gaussian Splatting avatar bound to the FLAME topology; the differentiable renderer outputs a 96x96 lip-region image through a virtual camera placed via reprojected landmarks. A frozen AutoAVSR model extracts visual speech features from both the rendered sequence and the ground-truth frames, and the cosine distance between them is backpropagated through the renderer to the vertex offsets. A three-stage curriculum—geometric pretraining on VOCASET, adaptation to MEAD pseudo-ground truth, and final joint training with $L_{\\text{read}}$—keeps the perceptual loss from destabilising early learning.","core_discovery":"The central claim is that perceptual supervision belongs in the rendered pixel domain, not on the intermediate mesh. VisualSpeaker renders predicted FLAME meshes through a per-subject 3D Gaussian Splatting avatar, crops a 96x96 lip region, and feeds those pixels to AutoAVSR, a pre-trained visual speech recognition model. The resulting lip-reading loss, $L_{\\text{read}} = 1 - \\text{CosSim}(\\text{AutoAVSR}(I_T), \\text{AutoAVSR}(\\hat{I}_T))$, measures whether the generated mouth motion is visually readable, and is added to a weighted vertex MSE in a final fine-tuning stage. The paper shows that the 3DGS render of the ground-truth mesh sits close to the real video in AutoAVSR's embedding space (cosine similarity 0.697 vs 0.190 for mismatched pairs), so the loss is a valid proxy for human lip reading. With this loss, the model produces crisper closures, more expressive large-scale lip motions, and distinct mouth shapes for minimal sign-language pairs, all while retaining mesh-driven controllability.","pith_inferences":["One could test whether the LVE gain comes from the perceptual loss itself or simply from the extra mouth-region attention: adding a mouth-weighted vertex loss with comparable weight in the final stage would isolate the perceptual contribution.","The embedding-alignment check (0.697 vs 0.190) was performed on renders of ground-truth meshes; monitoring the same similarity on renders of the model's own predictions during training would reveal whether the loss remains informative when meshes are imperfect.","The method suggests a general 'frozen 2D recognizer supervises 3D generation through a differentiable renderer' pattern that could extend to emotion, gaze, or identity perception, provided those models are also immune to the synthetic domain gap.","Since the loss is computed only on the lip crop, it ignores co-articulation cues above the mouth; a full-face render might capture more context, but the paper does not explore that variant."],"forward_implications":["Any mesh-driven talking-head pipeline can attach this render-level lip-reading loss to its existing vertex loss, since it only needs a differentiable head avatar and a frozen lip-reading model.","Text-to-mouthing becomes practical for sign-language avatars: distinct mouthings for minimal pairs such as 'why' and 'because' can be generated from glosses via a TTS model, with no paired audio or manual alignment.","Because the loss is defined on final pixels, future improvements in avatar rendering—finer tongues, inner-mouth geometry, better textures—automatically make the supervision stronger without changing the training objective.","The same recipe could supervise non-lip facial attributes by swapping the frozen perceptual model, but the paper does not test those cases.","The main practical cost is computational: differentiable rendering limits batch size to one with gradient accumulation, so the approach currently needs the cheap final-stage design the authors use."],"supporting_citations":[{"why":"Supplies the pre-trained AutoAVSR model whose visual speech features define the perceptual lip-reading loss.","marker":"[20]"},{"why":"The differentiable 3D Gaussian Splatting renderer that makes pixel-space supervision of the avatar geometry feasible.","marker":"[14]"},{"why":"Binds the 3DGS primitives to the FLAME mesh, letting predicted mesh deformations drive photorealistic renders.","marker":"[27]"},{"why":"FaceFormer transformer encoder-decoder architecture and training scheme that VisualSpeaker adapts.","marker":"[10]"},{"why":"VOCASET provides the high-quality 3D mesh ground truth for geometric pretraining.","marker":"[6]"},{"why":"MEAD provides the multi-view video dataset used for pseudo-ground-truth fitting, avatar creation, and evaluation.","marker":"[36]"},{"why":"Introduces the Lip Vertex Error metric used to report the 56.1% geometric improvement.","marker":"[29]"},{"why":"Prior mesh-level lip-reading guidance that VisualSpeaker extends by moving the loss to rendered pixels.","marker":"[9]"}],"fun_headline_variants":["Lip-reading AI supervises 3D avatar mouths, cuts error 56%","Perceptual lip-reading loss improves avatar lip sync by 56%","Visual speech recognizer grades avatar lip pixels, error down 56%","3D avatar lips refined by lip-reading loss: 56% lower vertex error","Lip-reader feedback teaches avatars to mouth words, 56% better"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pre-trained lip-reading model, trained on real video, will keep giving useful gradient information when watching this system's own synthetic renders of predicted meshes—especially in the early training stages when those meshes are rough.","fun_headline_variants_meta":{"raw":{"variants":["Lip-reading AI supervises 3D avatar mouths, cuts error 56%","Perceptual lip-reading loss improves avatar lip sync by 56%","Visual speech recognizer grades avatar lip pixels, error down 56%","3D avatar lips refined by lip-reading loss: 56% lower vertex error","Lip-reader feedback teaches avatars to mouth words, 56% better"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000906,"raw_usage":{"total_tokens":3896,"prompt_tokens":941,"completion_tokens":2955,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":557,"completion_tokens_details":{"reasoning_tokens":2855}},"tokens_in":557,"tokens_out":2955,"duration_ms":22406,"temperature":1.0,"reasoning_tokens":2855,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:12:11.232041+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the AutoAVSR cosine similarity between renders of the model's predicted meshes at an early epoch and their matching ground-truth frames. If the mean similarity of these matching pairs falls toward the mismatched-pair baseline of 0.190, the perceptual loss is not tracking lip articulation and the LVE improvement would not reflect true readability. A complementary test is a silent video lip-reading study pairing VisualSpeaker against the baseline at matched LVE; equal word accuracy would show the loss improves geometry but not perception.","supporting_citations":[{"cited_title":"Auto-A VSR: Audio-Visual Speech Recognition with Auto- matic Labels","cited_arxiv_id":null,"evidence_quote":"Supplies the pre-trained AutoAVSR model whose visual speech features define the perceptual lip-reading loss."},{"cited_title":"3D Gaussian Splatting for Real-Time Radiance Field Rendering.ACM Transactions on Graphics (TOG), 42(4), 2023","cited_arxiv_id":null,"evidence_quote":"The differentiable 3D Gaussian Splatting renderer that makes pixel-space supervision of the avatar geometry feasible."},{"cited_title":"Gaus- sianAvatars: Photorealistic Head Avatars with Rigged 3D Gaussians","cited_arxiv_id":null,"evidence_quote":"Binds the 3DGS primitives to the FLAME mesh, letting predicted mesh deformations drive photorealistic renders."},{"cited_title":"FaceFormer: Speech-Driven 3D Facial Ani- mation with Transformers","cited_arxiv_id":null,"evidence_quote":"FaceFormer transformer encoder-decoder architecture and training scheme that VisualSpeaker adapts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"VOCASET provides the high-quality 3D mesh ground truth for geometric pretraining."},{"cited_title":"MEAD: A Large-Scale Audio-Visual Dataset for Emo- tional Talking-Face Generation","cited_arxiv_id":null,"evidence_quote":"MEAD provides the multi-view video dataset used for pseudo-ground-truth fitting, avatar creation, and evaluation."},{"cited_title":"MeshTalk: 3D Face Animation from Speech Using Cross-Modality Disentangle- ment","cited_arxiv_id":null,"evidence_quote":"Introduces the Lip Vertex Error metric used to report the 56.1% geometric improvement."},{"cited_title":"Enhancing Speech-Driven 3D Facial Animation with Audio-Visual Guidance from Lip Reading Expert","cited_arxiv_id":"2407.01034","evidence_quote":"Prior mesh-level lip-reading guidance that VisualSpeaker extends by moving the loss to rendered pixels."}],"review_version":1}