Pith. sign in

REVIEW 4 major objections 5 minor 45 references

VisualSpeaker: Visually-Guided 3D Avatar Lip Synthesis

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read VisualSpeaker claims that supervising a 3D avatar's lip motions with a pre-trained lip-reading model, applied to photorealistic renders, cuts Lip Vertex Error by 56.1% on MEAD and improves perceived lip clarity.

desk verdict Sensible idea, but the headline 56.1% LVE improvement is confounded by extra training; the paper needs a proper control and baseline comparisons before that claim can be taken seriously. read the letter →

arxiv 2507.06060 v2 pith:DK6ASJDF submitted 2025-07-08 cs.CV cs.AI

Alexandre Symeonidis-Herzig , Özge Mercanoğlu Sincan , Richard Bowden This is my paper
classification cs.CVcs.AI
keywords 3Dfacialanimationlip-readinglossperceptualsupervisionGaussianSplattingvisualspeechrecognitionFLAMEmodelsignlanguagemouthinglipvertexerror
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes that a 3D talking avatar's mouth can be trained to be more accurate and more readable by having a pre-trained visual speech recogniser watch the rendered lips. The authors build VisualSpeaker, which predicts FLAME mesh deformations from audio or text, renders the mouth region as a photorealistic 3D Gaussian Splatting image, and adds a loss that pulls the recogniser's features for the render toward those of the ground-truth video. On the MEAD dataset, adding this lip-reading loss to a standard vertex loss lowers the Lip Vertex Error from 3.85 mm to 1.69 mm, a 56.1% reduction, and a user study prefers the resulting animations 65% of the time for realism and 66.6% for lip clarity. The motivation is that geometric losses alone reward averaged mouth shapes, whereas perceptual readability matters in applications like sign language, where mouthings disambiguate manual signs.

What carries the argument

The load-bearing object is the perceptual lip-reading loss computed on photorealistic renders. The pipeline first deforms a FLAME mesh with tooth geometry via an autoregressive transformer decoder, conditioned on Wav2Vec2.0 audio features or F5-TTS text-to-speech output. The predicted mesh drives a 3D Gaussian Splatting avatar bound to the FLAME topology; the differentiable renderer outputs a 96x96 lip-region image through a virtual camera placed via reprojected landmarks. A frozen AutoAVSR model extracts visual speech features from both the rendered sequence and the ground-truth frames, and the cosine distance between them is backpropagated through the renderer to the vertex offsets. A three-stage curriculum—geometric pretraining on VOCASET, adaptation to MEAD pseudo-ground truth, and final joint training with $L_{\text{read}}$—keeps the perceptual loss from destabilising early learning.

What would settle it

Measure the AutoAVSR cosine similarity between renders of the model's predicted meshes at an early epoch and their matching ground-truth frames. If the mean similarity of these matching pairs falls toward the mismatched-pair baseline of 0.190, the perceptual loss is not tracking lip articulation and the LVE improvement would not reflect true readability. A complementary test is a silent video lip-reading study pairing VisualSpeaker against the baseline at matched LVE; equal word accuracy would show the loss improves geometry but not perception.

Watch

Extended reading notes

Core claim

The central claim is that perceptual supervision belongs in the rendered pixel domain, not on the intermediate mesh. VisualSpeaker renders predicted FLAME meshes through a per-subject 3D Gaussian Splatting avatar, crops a 96x96 lip region, and feeds those pixels to AutoAVSR, a pre-trained visual speech recognition model. The resulting lip-reading loss, $L_{\text{read}} = 1 - \text{CosSim}(\text{AutoAVSR}(I_T), \text{AutoAVSR}(\hat{I}_T))$, measures whether the generated mouth motion is visually readable, and is added to a weighted vertex MSE in a final fine-tuning stage. The paper shows that the 3DGS render of the ground-truth mesh sits close to the real video in AutoAVSR's embedding space (cosine similarity 0.697 vs 0.190 for mismatched pairs), so the loss is a valid proxy for human lip reading. With this loss, the model produces crisper closures, more expressive large-scale lip motions, and distinct mouth shapes for minimal sign-language pairs, all while retaining mesh-driven controllability.

Load-bearing premise

The pre-trained lip-reading model, trained on real video, will keep giving useful gradient information when watching this system's own synthetic renders of predicted meshes—especially in the early training stages when those meshes are rough.

Editorial extensions

If this is right

  • Any mesh-driven talking-head pipeline can attach this render-level lip-reading loss to its existing vertex loss, since it only needs a differentiable head avatar and a frozen lip-reading model.
  • Text-to-mouthing becomes practical for sign-language avatars: distinct mouthings for minimal pairs such as 'why' and 'because' can be generated from glosses via a TTS model, with no paired audio or manual alignment.
  • Because the loss is defined on final pixels, future improvements in avatar rendering—finer tongues, inner-mouth geometry, better textures—automatically make the supervision stronger without changing the training objective.
  • The same recipe could supervise non-lip facial attributes by swapping the frozen perceptual model, but the paper does not test those cases.
  • The main practical cost is computational: differentiable rendering limits batch size to one with gradient accumulation, so the approach currently needs the cheap final-stage design the authors use.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One could test whether the LVE gain comes from the perceptual loss itself or simply from the extra mouth-region attention: adding a mouth-weighted vertex loss with comparable weight in the final stage would isolate the perceptual contribution.
  • The embedding-alignment check (0.697 vs 0.190) was performed on renders of ground-truth meshes; monitoring the same similarity on renders of the model's own predictions during training would reveal whether the loss remains informative when meshes are imperfect.
  • The method suggests a general 'frozen 2D recognizer supervises 3D generation through a differentiable renderer' pattern that could extend to emotion, gaze, or identity perception, provided those models are also immune to the synthetic domain gap.
  • Since the loss is computed only on the lip crop, it ignores co-articulation cues above the mouth; a full-face render might capture more context, but the paper does not explore that variant.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper proposes VisualSpeaker, an encoder-decoder transformer that predicts FLAME vertex offsets from audio or text and renders them through a per-subject 3D Gaussian Splatting avatar. During a third training stage, a perceptual lip-reading loss (Eq. 5) is computed by passing 96x96 lip crops of the rendered frames through a pretrained AutoAVSR model and comparing its embeddings with those of ground-truth video. The paper reports a 56.1% reduction in Lip Vertex Error on MEAD (Table 1), modest improvements in image metrics (Table 2), and a user study favoring the full model over the no-Lread baseline (Table 3). A text-to-mouthing application for BSL sign pairs is also demonstrated.

Significance. The core idea of supervising mesh-driven 3DGS avatars with a pretrained visual speech recognition model on rendered pixels is timely and potentially useful, especially for sign-language mouthing where lip readability is linguistically important. The paper is clearly written, the architecture is sensible, and the authors make a good-faith effort to validate the synthetic-to-real domain gap using an external embedding model and to support their claims with a user study. However, the central quantitative claim is currently undermined by a training-budget confound: the full model receives 100 additional epochs of MEAD training compared with the no-Lread baseline, and no Lvert-only control is reported. In addition, the paper compares only its own pipeline stages and not any published method. If the authors add the missing control and external comparisons, the contribution could become a solid systems paper; as it stands, the evidence does not yet establish that the lip-reading loss, rather than extra training, drives the reported improvements.

major comments (4)
  1. [Section 3.4, Table 1] The headline 56.1% LVE improvement is confounded by unequal training budgets. The baseline 'VisualSpeaker w/o Lread' is the stage-2 model trained for 250 epochs on MEAD with Lvert only, while the full model is trained for an additional 100 epochs with Lvert + lambda_read * Lread (batch size 1, gradient accumulation over 4). A control trained with Lvert alone for the same additional 100 epochs and identical schedule is not reported. Without this control, the LVE reduction from 3.85 mm to 1.69 mm cannot be unambiguously attributed to Lread; it may simply reflect continued optimization on the MEAD training set. This is load-bearing for the abstract's central claim, so the authors should add an Lvert-only fine-tuning control and report its LVE.
  2. [Section 4.2, Table 1] The paper compares only its own successive pipeline stages and no published method, yet the introduction and conclusion claim that VisualSpeaker surpasses mesh-based baselines. No LVE numbers are reported for FaceFormer, CodeTalker, EMOTE, GaussianTalker, or other prior work on the same MEAD test split and pseudo-ground-truth fitting. Because LVE is a standard metric in this area, adding comparisons to at least one or two representative methods under the same evaluation protocol is necessary to support the claimed geometric improvement over the state of the art.
  3. [Section 3.2] The domain-gap validation is performed only on renders of ground-truth meshes: the authors report cosine similarity 0.697 for matching pairs and 0.190 for mismatched pairs. During stage 3, however, Lread is applied to renders of predicted meshes, which are of lower quality early in training. The paper does not measure whether AutoAVSR embeddings remain aligned for such predicted-mesh renders, so it is possible that Lread rewards spurious visual features rather than lip articulation during the early part of the final stage. The authors should compute the same cosine-similarity statistic on predicted-mesh renders at representative points during training, or otherwise demonstrate that the loss gradients are not chasing out-of-distribution rendering artifacts.
  4. [Section 4.3 and Supplementary 6.2] The user study inherits the same training-budget confound, because 'Ours vs. Baseline' compares the stage-3 model against the stage-2 model. Moreover, the study instructions state that 'Audio is provided with the videos,' so the 'lip clarity' ratings may reflect audio-visual synchronization and overall intelligibility rather than pure visual lip-readability. To support the claim that the perceptual loss improves lip readability, the authors should run a silent (audio-free) condition or at least report results broken down by audio-present vs audio-absent presentation.
minor comments (5)
  1. [Section 3.3, Eq. (4)] The symbol V is used for both the total vertex count and the vertex index; please rename one of them for clarity.
  2. [Section 4.2] The sentence 'is an key metric' contains a typo and should read 'is a key metric.'
  3. [Section 4.2] The phrase 'the retrained models performance on VOCASET' should be 'the pretrained model's performance on VOCASET.'
  4. [Section 4.3] The word 'ingrate' should be 'integrate' in the final sentence of Section 4.3.
  5. [Supplementary 6.2] The description of how categorical preferences are converted to a {−2,...,2} score and then to 'percentage of times preferred' is underspecified; please clarify how ties and unequal numbers of raters per video are handled.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the lip-reading loss is an externally pretrained visual-speech model applied to rendered pixels; the headline LVE gain is an empirical training result, not a definitional reduction.

full rationale

The claimed derivation is the perceptual loss L_read = 1 - CosSim(AutoAVSR(I_T), AutoAVSR(^I_T)) (Eq. 5), where AutoAVSR is a pretrained external model [20] and I_T and ^I_T are ground-truth frames and differentiable 3DGS renders of predicted meshes. None of these components is defined in terms of the reported target metric: LVE is computed on FLAME lip vertices (Section 4.2), while L_read acts on 96x96 rendered lip pixel crops through a frozen external V-ASR encoder, so minimizing Eq. 5 cannot reduce to Eq. 4 by construction. The domain-gap check (cosine similarity 0.697 for matching pairs vs 0.190 for mismatched pairs) is an external validation against real video, not a self-referential fit. The only hand-tuned scalar, lambda_read = 1e-5, is an optimization weight chosen empirically, with the paper reporting that lower values yield negligible improvements and higher values cause artifacts; it is not a fitted parameter later renamed as a prediction. Citations to VHAP [26] and GaussianAvatars [27] are external method choices, not self-citations, and no uniqueness theorem or prior-work ansatz is invoked to force the architecture. The paper itself acknowledges limitations, including under-articulation of rapid or subtle consonant closures, which is consistent with an honest empirical evaluation rather than a circular claim. The most serious threat to the central quantitative claim is the experimental confound that the full model receives 100 additional MEAD epochs relative to the 'w/o L_read' baseline, so the 56.1% LVE improvement is not unambiguously attributable to L_read; however, this is a training-protocol confound and an evaluation-validity concern, not circular reasoning in the derivation chain. Under the hard rules, confounding belongs in correctness risk rather than in a circularity score, and the paper's loss is self-contained against an external benchmark.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The central claim depends on three domain assumptions: the pseudo-GT fit quality, the AutoAVSR embedding as a perceptual proxy, and the render-to-real domain gap. All are acknowledged or partially validated, but none is proven. The only direct free parameter of the new loss is lambda_read, but the curriculum and vertex weights also shape the result.

free parameters (4)
  • lambda_read = 1e-5
    Weight of the lip-reading loss in Eq. (6), chosen empirically in Section 3.4; 1e-6 gave negligible gains and higher values caused mesh artifacts.
  • per-vertex weights W_v = 1.0 (stage 1), 0.5 non-skin, 0.0 eyes (stages 2-3)
    Manual weights in Eq. (4) to counter pseudo-GT noise and eye motion; directly affect the dominant loss in early training.
  • training schedule hyperparameters = 250/100 epochs, batch size 4/1 with 4-step accumulation, lr 1e-4
    Hand-picked in Section 4, not swept; the two-stage curriculum is load-bearing for stability.
  • decoder dimensions = 64-dim vertex embedding, 4 heads, dropout 0.3
    Architecture choices in Section 3.3; no ablation shows their effect.
assumptions (6)
  • domain assumption AutoAVSR embedding space is a valid proxy for human lip readability
    Eq. (5) minimizes cosine distance in this space. Section 3.2 reports GT vs render cosine similarity 0.697 vs 0.190 mismatched, but that does not prove optimizing this metric improves human lip-reading.
  • domain assumption 3DGS renders of predicted meshes stay within the AutoAVSR training distribution throughout training
    The domain-gap check uses renders of ground-truth meshes; early training renders of predicted meshes are likely lower quality, and this transfer is assumed.
  • domain assumption FLAME with 120 added teeth triangles captures articulatory detail needed for lip-reading
    No explicit tongue or inner-mouth geometry; Section 4.3 admits under-articulation of plosives and tongue contacts.
  • domain assumption Pseudo-ground-truth FLAME fits from MEAD are accurate enough for training and LVE evaluation
    Section 3.1 says the fits produce noisier GT than direct scans; LVE is measured against these noisy meshes.
  • domain assumption Wav2Vec2 and F5-TTS provide sufficient audio/text representations for animation
    The encoder uses frozen Wav2Vec2 features and F5-TTS synthesized audio; the impact of TTS errors on lip accuracy is not evaluated.
  • standard math Standard transformer, MSE, and cosine similarity mathematics
    Backpropagation through attention, MSE, and cosine similarity are standard.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VisualSpeaker: Visually-Guided 3D Avatar Lip Synthesis." pith.science (2026). https://pith.science/paper/DK6ASJDF

@misc{pith2026250706060,
  author       = {Pith},
  title        = {Pith review of: VisualSpeaker: Visually-Guided 3D Avatar Lip Synthesis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DK6ASJDF}},
  note         = {Machine review of arXiv:2507.06060}
}
read the original abstract

Realistic, high-fidelity 3D facial animations are crucial for expressive avatar systems in human-computer interaction and accessibility. Although prior methods show promising quality, their reliance on the mesh domain limits their ability to fully leverage the rapid visual innovations seen in 2D computer vision and graphics. We propose VisualSpeaker, a novel method that bridges this gap using photorealistic differentiable rendering, supervised by visual speech recognition, for improved 3D facial animation. Our contribution is a perceptual lip-reading loss, derived by passing photorealistic 3D Gaussian Splatting avatar renders through a pre-trained Visual Automatic Speech Recognition model during training. Evaluation on the MEAD dataset demonstrates that VisualSpeaker improves both the standard Lip Vertex Error metric by 56.1% and the perceptual quality of the generated animations, while retaining the controllability of mesh-driven animation. This perceptual focus naturally supports accurate mouthings, essential cues that disambiguate similar manual signs in sign language avatars.

Figures

Figures reproduced from arXiv: 2507.06060 by the authors.

Figure 1
Figure 1. VisualSpeaker results. The generated animation from the phrase “No price too high.” Ground truth video (top), the FLAME meshes predicted by our approach (middle), and the 3DGS renders driven by these meshes (bottom). Note how VisualSpeaker synthesizes lip movements that accurately and expressively articulate the input, achieved by combining geometric and perceptual supervision. Abstract Realistic, high-fidelity 3D f… view at source ↗
Figure 2
Figure 2. Lip Region Comparison. Visual comparison of lip re￾gions after alignment, cropping, and grayscale conversion for lip￾reading supervision. The 3DGS render (middle) closely resembles the ground truth (left), while the mesh render (right) lacks photo￾realistic detail. 3 [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Overview of VisualSpeaker. Our encoder–decoder framework predicts the next frame’s vertex offsets, Vt. Given either text or audio, the encoder (left) generates input features IT . These, together with past facial motion, V<t, and a speaker identity embedding, sn, derived from a neutral FLAME mesh, are passed to the decoder (middle). During training (right), predictions are supervised by a standard vertex loss and a … view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Qualitative Results. Visual comparisons for four unseen subjects and sentences from MEAD [36], highlighting how VisualSpeaker better preserves lip articulation than the baseline. Each subfigure displays three frames, left to right: Pseudo-GT ren￾der, VisualSpeaker with…
Figure 5
Figure 5. Figure 5: Qualitative Comparison. Example outputs from VisualSpeaker and the model pretrained only on VOCASET, on unseen subjects and sentences from MEAD [36]. Despite the clear perceptual differences, the PSNR values for these frames are 21.11 dB and 20.90 dB, respectively. 4.3…
Figure 6
Figure 6. Figure 6: SLP Example. Two left sub-figures show still [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Cosine similarity confusion matrix. 6.2. User Study Details The user study consisted of 51 participants, each evaluating 20 pairs of videos. They were recruited via a departmen￾tal mailing list. We circulated 5 variants of the study, each with a different set of 20 pai…
Figure 8
Figure 8. Figure 8: Sample User Study Interface. 1 [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

45 extracted references · 40 canonical work pages

  1. [1]

    wav2vec 2.0: A Framework for Self- Supervised Learning of Speech Representations.Advances in Neural Information Processing Systems, 33:12449–12460,

    Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A Framework for Self- Supervised Learning of Speech Representations.Advances in Neural Information Processing Systems, 33:12449–12460,

    work page

  2. [2]

    A Morphable Model for the Synthesis of 3D Faces

    V olker Blanz and Thomas Vetter. A Morphable Model for the Synthesis of 3D Faces. InProceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH), pages 187–194, 1999. 3

    work page 1999

  3. [3]

    Video Rewrite: Driving Visual Speech with Audio, pages 353–360

    Christoph Bregler, Michele Covell, and Malcolm Slaney. Video Rewrite: Driving Visual Speech with Audio, pages 353–360. 1997. 2

    work page 1997

  4. [4]

    Perceptually Accu- rate 3D Talking Head Generation: New Definitions, Speech- Mesh Representation, and Evaluation Metrics

    Lee Chae-Yeon, Oh Hyun-Bin, Han EunGi, Kim Sung-Bin, Suekyeong Nam, and Tae-Hyun Oh. Perceptually Accu- rate 3D Talking Head Generation: New Definitions, Speech- Mesh Representation, and Evaluation Metrics. InProceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 21065–21074, 2025. 2

    work page 2025

  5. [5]

    Cohen and Dominic W

    Michael M. Cohen and Dominic W. Massaro. Synthesis of a more natural-sounding talking head.Behavior Research Methods, Instruments, & Computers, 22(2):260–263, 1990. 2

    work page 1990

  6. [6]

    Daniel Cudeiro, Timo Bolkart, Cassidy Laidlaw, Anurag Ranjan, and Michael J. Black. VOCA: V oice Operated Char- acter Animation. InProceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), pages 10101–10111, 2019. 1, 2, 5, 6

    work page 2019

  7. [7]

    EMOTE: Emo- tional Speech-Driven Animation with Content-Emotion Dis- entanglement

    Radek Dan ˇeˇcek, Kiran Chhatre, Shashank Tripathi, Yandong Wen, Michael Black, and Timo Bolkart. EMOTE: Emo- tional Speech-Driven Animation with Content-Emotion Dis- entanglement. InSIGGRAPH Asia 2023 Conference Papers, pages 1–13, 2023. 2, 3

    work page 2023

  8. [8]

    JALI: An Animator-Centric Viseme Model for Ex- pressive Lip-Synchronization.ACM Transactions on Graph- ics (TOG), 35(4):1–11, 2016

    Pif Edwards, Chris Landreth, Eugene Fiume, and Karan Singh. JALI: An Animator-Centric Viseme Model for Ex- pressive Lip-Synchronization.ACM Transactions on Graph- ics (TOG), 35(4):1–11, 2016. 2

    work page 2016

Show all 45 references
  1. [9]

    Enhancing Speech-Driven 3D Facial Animation with Audio-Visual Guidance from a Lip Reading Expert.arXiv preprint arXiv:2407.01034, 2024

    Han EunGi, Oh Hyun-Bin, Kim Sung-Bin, Corentin Nivelet Etcheberry, Suekyeong Nam, Janghoon Joo, and Tae-Hyun Oh. Enhancing Speech-Driven 3D Facial Animation with Audio-Visual Guidance from a Lip Reading Expert.arXiv preprint arXiv:2407.01034, 2024. 2

    2024 arXiv

  2. [10]

    FaceFormer: Speech-Driven 3D Facial Ani- mation with Transformers

    Yingruo Fan, Zhaojiang Lin, Jun Saito, Wenping Wang, and Taku Komura. FaceFormer: Speech-Driven 3D Facial Ani- mation with Transformers. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 1, 2, 4, 5

    2022

  3. [11]

    Confusions among Visually Perceived Con- sonants.Journal of Speech, Language, and Hearing Re- search, 11(4):796–804, 1968

    Cletus G Fisher. Confusions among Visually Perceived Con- sonants.Journal of Speech, Language, and Hearing Re- search, 11(4):796–804, 1968. 2

    1968

  4. [12]

    AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis

    Yudong Guo, Keyu Chen, Sen Liang, Yong-Jin Liu, Hujun Bao, and Juyong Zhang. AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis. InProceedings of the IEEE/CVF International Conference on Computer Vi- sion (ICCV), pages 5784–5794, 2021. 2

    2021

  5. [13]

    Audio-Driven Facial Animation by Joint End-to-End Learning of Pose and Emotion.ACM Transac- tions on Graphics (TOG), 36(4):1–12, 2017

    Tero Karras, Timo Aila, Samuli Laine, Antti Herva, and Jaakko Lehtinen. Audio-Driven Facial Animation by Joint End-to-End Learning of Pose and Emotion.ACM Transac- tions on Graphics (TOG), 36(4):1–12, 2017. 2

    2017

  6. [14]

    3D Gaussian Splatting for Real-Time Radiance Field Rendering.ACM Transactions on Graphics (TOG), 42(4), 2023

    Bernhard Kerbl, Georgios Kopanas, Thomas Leimk ¨uhler, and George Drettakis. 3D Gaussian Splatting for Real-Time Radiance Field Rendering.ACM Transactions on Graphics (TOG), 42(4), 2023. 2, 3

    2023

  7. [15]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization.arXiv preprint arXiv:1412.6980,

    arXiv

  8. [16]

    Avat3r: Large An- imatable Gaussian Reconstruction Model for High-Fidelity 3D Head Avatars.arXiv preprint arXiv:2502.20220, 2025

    Tobias Kirschstein, Javier Romero, Artem Sevastopolsky, Matthias Nießner, and Shunsuke Saito. Avat3r: Large An- imatable Gaussian Reconstruction Model for High-Fidelity 3D Head Avatars.arXiv preprint arXiv:2502.20220, 2025. 3

    2025

  9. [17]

    S3D-NeRF: Single-Shot Speech-Driven Neural Radiance Field for High Fidelity Talking Head Synthesis

    Dongze Li, Kang Zhao, Wei Wang, Yifeng Ma, Bo Peng, Yingya Zhang, and Jing Dong. S3D-NeRF: Single-Shot Speech-Driven Neural Radiance Field for High Fidelity Talking Head Synthesis. InProceedings of the European Conference on Computer Vision (ECCV), pages 365–382,

  10. [18]

    TalkingGaussian: Structure-Persistent 3D Talking Head Synthesis via Gaussian Splatting

    Jiahe Li, Jiawei Zhang, Xiao Bai, Jin Zheng, Xin Ning, Jun Zhou, and Lin Gu. TalkingGaussian: Structure-Persistent 3D Talking Head Synthesis via Gaussian Splatting. InPro- ceedings of the European Conference on Computer Vision (ECCV), pages 127–145, 2024. 2

    2024

  11. [19]

    Black, Hao Li, and Javier Romero

    Tianye Li, Timo Bolkart, Michael J. Black, Hao Li, and Javier Romero. Learning a Model of Facial Shape and Ex- pression from 4D Scans.ACM Transactions on Graphics (TOG), 36(6):1–17, 2017. 3

    2017

  12. [20]

    Auto-A VSR: Audio-Visual Speech Recognition with Auto- matic Labels

    Pingchuan Ma, Alexandros Haliassos, Adriana Fernandez- Lopez, Honglie Chen, Stavros Petridis, and Maja Pantic. Auto-A VSR: Audio-Visual Speech Recognition with Auto- matic Labels. InProceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (IC...

    2023

  13. [21]

    Picture My V oice: Audio to Visual Speech Synthesis Using Artificial Neural Networks

    Dominic W Massaro, Jonas Beskow, Michael M Cohen, Christopher L Fry, and Tony Rodriguez. Picture My V oice: Audio to Visual Speech Synthesis Using Artificial Neural Networks. InProceedings of the International Conference on Auditory-Visual Speech Processing (AVSP), pages 133– ...

    1999

  14. [22]

    Real-Time Talking Head Driven by V oice and Its Application to Communication and Entertain- ment

    Shigeo Morishima. Real-Time Talking Head Driven by V oice and Its Application to Communication and Entertain- ment. InProceedings of the International Conference on Auditory-Visual Speech Processing (AVSP), pages 195–200,

  15. [23]

    FaceFX, 2025

    OC3 Entertainment. FaceFX, 2025. 2

    2025

  16. [24]

    Filntisis, Petros Maragos, and Anastasios Roussos

    Foivos Paraperas Papantoniou, Panagiotis P. Filntisis, Petros Maragos, and Anastasios Roussos. Neural Emotion Director: Speech-Preserving Semantic Control of Facial Expressions in ”In-the-Wild” Videos. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Rec...

    2022

  17. [25]

    Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black. Expressive Body Capture: 3D Hands, Face, and Body from a Single Image. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni...

    2019

  18. [26]

    VHAP: Versatile Head Alignment with Adaptive Appearance Priors.https://github.com/ ShenhanQian/VHAP, 2024

    Shenhan Qian. VHAP: Versatile Head Alignment with Adaptive Appearance Priors.https://github.com/ ShenhanQian/VHAP, 2024. 3

    2024

  19. [27]

    Gaus- sianAvatars: Photorealistic Head Avatars with Rigged 3D Gaussians

    Shenhan Qian, Tobias Kirschstein, Liam Schoneveld, Davide Davoli, Simon Giebenhain, and Matthias Nießner. Gaus- sianAvatars: Photorealistic Head Avatars with Rigged 3D Gaussians. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages ...

    2024

  20. [28]

    Accelerating 3D Deep Learning with PyTorch3D.arXiv preprint arXiv:2007.08501, 2020

    Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan-Yen Lo, Justin Johnson, and Georgia Gkioxari. Accelerating 3D Deep Learning with PyTorch3D.arXiv preprint arXiv:2007.08501, 2020. 3

    2007 arXiv

  21. [29]

    MeshTalk: 3D Face Animation from Speech Using Cross-Modality Disentangle- ment

    Alexander Richard, Michael Zollh ¨ofer, Yandong Wen, Fer- nando de la Torre, and Yaser Sheikh. MeshTalk: 3D Face Animation from Speech Using Cross-Modality Disentangle- ment. InProceedings of the IEEE/CVF International Confer- ence on Computer Vision (ICCV), pages 1173–1182, 2...

    2021

  22. [30]

    Sch ¨onberger and Jan-Michael Frahm

    Johannes L. Sch ¨onberger and Jan-Michael Frahm. Structure- from-Motion Revisited. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4104–4113, 2016. 3

    2016

  23. [31]

    Neural Haircut: Prior-Guided Strand-Based Hair Reconstruction

    Vanessa Sklyarova, Jenya Chelishev, Andreea Dogaru, Igor Medvedev, Victor Lempitsky, and Egor Zakharov. Neural Haircut: Prior-Guided Strand-Based Hair Reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 19762–19773, 2023. 3

    2023

  24. [32]

    Cambridge Univer- sity Press, 1999

    Rachel Sutton-Spence and Bencie Woll.The Linguistics of British Sign Language: An Introduction. Cambridge Univer- sity Press, 1999. 1, 2

    1999

  25. [33]

    A Deep Learning Approach for Gener- alized Speech Animation.ACM Transactions on Graphics (TOG), 36(4):1–11, 2017

    Sarah Taylor, Taehwan Kim, Yisong Yue, Moshe Mahler, James Krahe, Anastasio Garcia Rodriguez, Jessica Hodgins, and Iain Matthews. A Deep Learning Approach for Gener- alized Speech Animation.ACM Transactions on Graphics (TOG), 36(4):1–11, 2017. 2

    2017

  26. [34]

    Neural Discrete Representation Learning.Advances in Neural Information Processing Systems, 30, 2017

    Aaron Van Den Oord and Oriol Vinyals. Neural Discrete Representation Learning.Advances in Neural Information Processing Systems, 30, 2017. 2

    2017

  27. [35]

    Attention Is All You Need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. InAdvances in Neu- ral Information Processing Systems, 2017. 2

    2017

  28. [36]

    MEAD: A Large-Scale Audio-Visual Dataset for Emo- tional Talking-Face Generation

    Kaisiyuan Wang, Qianyi Wu, Linsen Song, Zhuoqian Yang, Wayne Wu, Chen Qian, Ran He, Yu Qiao, and Chen Change Loy. MEAD: A Large-Scale Audio-Visual Dataset for Emo- tional Talking-Face Generation. InProceedings of the Euro- pean Conference on Computer Vision (ECCV), 2020. 3, 5, 6, 7

    2020

  29. [37]

    Bovik, Hamid R

    Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. Image Quality Assessment: From Error Visi- bility to Structural Similarity.IEEE Transactions on Image Processing, 13(4):600–612, 2004. 7

    2004

  30. [38]

    CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Prior

    Jinbo Xing, Menghan Xia, Yuechen Zhang, Xiaodong Cun, Jue Wang, and Tien-Tsin Wong. CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Prior. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12780–12790, 2023. 1, 2

    2023

  31. [39]

    GaussianTalker: Speaker-Specific Talking Head Synthesis via 3D Gaussian Splatting

    Hongyun Yu, Zhan Qu, Qihang Yu, Jianchuan Chen, Zhonghua Jiang, Zhiwen Chen, Shengyu Zhang, Jimin Xu, Fei Wu, Chengfei Lv, and Gang Yu. GaussianTalker: Speaker-Specific Talking Head Synthesis via 3D Gaussian Splatting. InProceedings of the 32nd ACM International Conference on ...

    2024

  32. [40]

    GUA V A: Gener- alizable Upper-Body 3D Gaussian Avatars.arXiv preprint arXiv:2505.03351, 2025

    Dongbin Zhang, Yunfei Liu, Lijian Lin, Ye Zhu, Yang Li, Minghan Qin, Yu Li, and Haoqian Wang. GUA V A: Gener- alizable Upper-Body 3D Gaussian Avatars.arXiv preprint arXiv:2505.03351, 2025. 3

    2025 arXiv

  33. [41]

    The Unreasonable Effectiveness of Deep Features as a Perceptual Metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shecht- man, and Oliver Wang. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 586–595, 2018. 7

    2018

  34. [42]

    SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation

    Wenxuan Zhang, Xiaodong Cun, Xuan Wang, Yong Zhang, Xi Shen, Yu Guo, Ying Shan, and Fei Wang. SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and P...

    2023

  35. [43]

    HeadGaP: Few-Shot 3D Head Avatar via Generalizable Gaussian Priors.arXiv preprint arXiv:2408.06019, 2024

    Xiaozheng Zheng, Chao Wen, Zhaohu Li, Weiyi Zhang, Zhuo Su, Xu Chang, Yang Zhao, Zheng Lv, Xiaoyuan Zhang, Yongjie Zhang, et al. HeadGaP: Few-Shot 3D Head Avatar via Generalizable Gaussian Priors.arXiv preprint arXiv:2408.06019, 2024. 3

    2024 arXiv

  36. [44]

    Learn2Talk: Learning to Talk and Listen from 2D and 3D Talking Faces.IEEE Transactions on Visualiza- tion and Computer Graphics, 2024

    Yixiang Zhuang, Baoping Cheng, Yao Cheng, Yuntao Jin, Renshuai Liu, Chengyang Li, Xuan Cheng, Jing Liao, and Juncong Lin. Learn2Talk: Learning to Talk and Listen from 2D and 3D Talking Faces.IEEE Transactions on Visualiza- tion and Computer Graphics, 2024. 2, 3 10 VisualSpeake...

    2024

  37. [45]

    Supplementary Material 6.1. AutoA VSR Feature Alignment The confusion matrix in Figure 7 shows how cosine sim- ilarity scores are strongest along the diagonal, with non- matching videos elsewhere scoring far lower. This indicates that the features of the 3DGS render closely ma...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.