Pith. sign in

REVIEW 3 major objections 6 minor 110 references

A single-identity 3D Gaussian proxy avatar, retargeted to arbitrary portraits and accelerated by low-rank appearance caching, achieves real-time emotion-controllable portrait animation at 32 FPS.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A cascade of a single-identity Gaussian emotion proxy, a one-shot diffusion retargeting model, and low-rank appearance caching enables real-time one-shot portrait animation with emotion control.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection A coherent cascaded system with two real ideas — single-identity Gaussian proxy and low-rank appearance caching — but the missing appendix leaves the cross-identity claim under-supported. the 3 major comments →

arxiv 2608.01978 v1 pith:P3SUFYTL submitted 2026-08-03 cs.CV

Proxy Avatar Meets Low-Rank Caching: Real-Time One-Shot Emotion-Controllable Portrait Animation

Haijie Yang , Jindi Bao , Yixuan Dong , Hongliang Zhang , Jian Bi , Hao Tang , Zhenyu Zhang , Jianjun Qian
show 1 more author
Jian Yang
This is my paper
classification cs.CV
keywords portrait animationaudio-driven talking heademotion controlone-shot generation3D Gaussian proxy avatarlow-rank cachingdiffusion distillationreal-time inference
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes a three-stage pipeline to prove that real-time one-shot emotion-controllable portrait animation can be achieved without forcing a single model to learn emotion, identity, and efficiency at once. A 3D Gaussian proxy avatar, trained once on one subject, converts audio and an emotion label into an expressive driving video; this video is used only as a motion carrier, not as the target appearance. A large-scale one-shot retargeting model then extracts identity-independent motion from the proxy video and adapts it to any reference portrait. To make diffusion-based synthesis fast enough, the paper reuses the reference appearance features taken at the first denoising step and approximates their later step-to-step variations with low-rank residual matrices. If the claim holds, users could animate a single still portrait with controllable emotions and accurate lip sync in real time on a consumer GPU, without per-identity training or reference emotion videos.

Core claim

The central discovery is that the emotional motion prior can be separated from the target identity by learning it once in a Gaussian proxy-avatar space. The proxy avatar predicts, for each frame, a neutral speech motion plus an emotion-dependent residual offset added to Gaussian position, rotation, scale, color, and opacity; the resulting video is then treated as an identity-independent driving signal by a one-shot diffusion model that injects the motion through implicit motion-attention and explicit pose keypoints. To meet the real-time goal, the appearance branch is computed in full only at the first distilled denoising step, and each later step conditions on that cached feature plus a low

What carries the argument

The load-bearing object is the 3D Gaussian emotion proxy avatar: a set of 3D Gaussians (position, rotation, scale, color, opacity) whose attributes are offset by MLP regressors fed by a spatial-audio-emotion joint attention module. The motion is decomposed into neutral speech motion and emotion-residual offsets so the emotion branch focuses on subtle facial dynamics rather than re-learning lip motion. The second named mechanism is low-rank appearance caching: after a few-step distillation schedule, the full ReferenceNet appearance feature is computed once at the first denoising step, and later steps approximate it as F1 + AjBj with light learnable low-rank matrices, supervised by an appearan

Load-bearing premise

The load-bearing premise is that a single-identity Gaussian proxy avatar, trained once on one actor's expressive speech, provides emotion-aware motion priors general enough to drive arbitrary target portraits after the one-shot retargeting step; the paper states in its experimental section that an appendix analysis of actor-specific motion bias is provided, but that appendix is not present in the submitted text, so the generalization claim rests on the qualitative and quantit

What would settle it

Train the proxy avatar on a second, visually distinct actor and drive the same reference portraits with identical audio and emotion labels. If the resulting emotion expressiveness, identity preservation, or lip-sync scores drop materially, the proxy's motion is tied to the actor's idiosyncrasies and the central generalization claim fails. A second falsifier: compute the actual feature difference between the ReferenceNet output at the first and later denoising steps; if the low-rank residual AjBj cannot approximate that difference within a small error bound on a held-out set, the caching approx

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Emotion-controllable portrait animation becomes a real-time application: a consumer GPU can render 32 FPS talking heads from a single reference image, audio, and an emotion label.
  • The decoupling of motion and appearance means the expensive motion prior is trained once and reused; new target identities require no fine-tuning and no emotion-specific reference videos.
  • Low-rank appearance caching is a generally applicable speed-up for diffusion-based portrait animation, since a fixed reference image's appearance features are highly reusable across denoising steps.
  • In the paper's comparison, the framework outperforms existing emotion-controllable methods on cross-identity emotion expressiveness, identity preservation, and lip synchronization, suggesting that single-identity training can transfer emotional dynamics across faces.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the single-identity proxy claim generalizes, the same division of labor could be applied to other continuous motion signals—gaze, head gestures, body poses—by training one 'motion actor' and retargeting it to any appearance.
  • The proxy-avatar design could support a library of motion priors (e.g., one avatar per emotion intensity or speaking style) selected at inference time, without retraining the retargeting module.
  • A strong, direct test of the identity-independence claim is to train the proxy on several different actors and drive the same target portraits: if the retargeting truly discards proxy identity, emotion transfer quality should be unchanged across proxies.
  • The appearance-caching trick could be extended to other fixed conditioning inputs in diffusion models (depth, edge maps, text embeddings) when their per-step variation is small and low-rank, though the paper does not test those cases.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes a three-stage pipeline for real-time one-shot emotion-controllable portrait animation. Stage 1 builds a 3D Gaussian proxy avatar trained on a single MEAD identity, which renders an emotional driving video from audio and an emotion label. Stage 2 uses a one-shot diffusion retargeting model to extract identity-independent motion and pose from the proxy frames and transfer them to an arbitrary reference portrait. Stage 3 introduces appearance redundancy distillation: the reference appearance feature is computed once at the first denoising step, and per-step variations are approximated by low-rank adapters. Experiments compare against recent baselines on self-reconstruction and cross-identity driving, reporting 32 FPS on an RTX 4090 and state-of-the-art emotion accuracy, identity similarity, and lip sync in the cross-identity table.

Significance. The division-of-labor design is conceptually attractive: a single reusable Gaussian proxy for emotional motion, a large-scale retargeting bridge, and a low-rank caching mechanism for efficiency. If validated, the framework could reduce the cost of emotion-controllable animation while preserving one-shot generalization. The paper gives clear ablations that isolate the proxy-avatar and low-rank-caching contributions, and the cross-identity metrics (FID, ID-Sim, Emo-Acc, SyncNet) are appropriate for the unpaired setting. However, the central claims currently rest on an appendix that is not included and on an untested domain gap between the real-video training distribution of the retargeting encoders and the synthetic Gaussian-splat proxy frames used at inference. These are load-bearing, and the paper is not publishable in its current form without addressing them.

major comments (3)
  1. [§3.2, Eqs. (13)–(18)] The one-shot driving module is trained on natural video corpora (VFHQ, NerSemble, DH-FaceVid-1K), but at inference the driving frames Vp are rendered by a 3D Gaussian proxy trained on a single MEAD subject (§3.1, Eq. (10)). The motion encoder Em and the keypoint encoder Ek have never been trained or fine-tuned on splat-rendered images. If proxy renderings lie outside the training distribution, the extracted motion cue cmot_t (Eq. 14) and the target-space keypoints kd_t (Eq. 17) are corrupted, directly undermining the cross-identity results in Table 2. I request a control experiment: drive the same reference portrait with (a) the real video of the proxy identity for the same audio/emotion clip and (b) the proxy-rendered version of that clip, and report Emo-Acc, ID-Sim, and Lip Sync for both. Also report the feature-space distance between Em(proxy frame) and Em(real frame).
  2. [§3.1–§3.3 and Experiments] The manuscript repeatedly defers essential technical content to an appendix that is not provided: the proxy-avatar training objectives (§3.1), the motion-driving module losses (§3.2), the step-distillation loss Lstep (§3.3, Eq. (27)), and the promised analysis of actor-specific motion bias, storage overhead, and cross-identity generalization. Without the actual loss definitions, the identity-independent motion objective cannot be assessed, and the effect of λapp in Eq. (26) is unverifiable. More importantly, the paper's weakest assumption—that a single-identity proxy generalizes to arbitrary target portraits—has no supporting evidence because the promised appendix analysis is absent. The appendix must be included in the submission.
  3. [Table 2 and Fig. 1] The headline real-time claim is presented visually in Fig. 1, but the FPS bar chart omits EmoTag, which Table 1 reports as 45 FPS—higher than the proposed 32 FPS. The speed advantage is demonstrated mainly against the paper's own unaccelerated variants (17.8/21.5 FPS), not against the fastest baseline. Please include all Table 1 baselines in Fig. 1 and discuss the quality/speed tradeoff explicitly. In addition, Table 2 reports only point estimates over 10 randomly selected target identities, with no error bars or significance tests; the Emo-Acc metric is not tied to a named emotion classifier. Since Table 2 is the main support for the cross-identity claims, report per-clip variability or confidence intervals and specify the classifier.
minor comments (6)
  1. [Eq. (22)] The index in the residual approximation is written as i=2,...,K but the expression uses eF_j and ΔF_j; the index should be j.
  2. [Fig. 2] Typo: 'Low-Rnak' should be 'Low-Rank'.
  3. [Throughout] The baseline name is inconsistently spelled 'EmoTaG' in some places and 'EmoTag' in others; please unify.
  4. [Implementation Details] The text says 'denoising steps are set to N=4' immediately before defining K ≪ N in Eq. (19). Clarify whether N is the original step count or the distilled step count, and define the relationship to K.
  5. [Implementation Details] The Historical Keyframe Mechanism (HKM) is mentioned with a threshold τ=17 but is never introduced or described in the paper. Add a definition or remove the reference.
  6. [References] There are two identical entries for Wang et al. 2004 with the same title; one should be removed or differentiated.

Circularity Check

0 steps flagged

No circular derivation: each stage is trained against external data or an external teacher, and the central claims are evaluated independently.

full rationale

The paper's derivation chain is self-contained with respect to circularity. Stage 1 (3D Gaussian emotion proxy avatar) is trained on MEAD ground-truth video conditioned on audio and emotion labels; its output is a rendered driving video, not the target portrait. Stage 2 (one-shot portrait motion driving) is trained on external real-video datasets (VFHQ, NerSemble, DH-FaceVid-1K) and receives the proxy video as an input signal; no equation defines the retargeted motion as a fitted version of the proxy output. Stage 3 (low-rank caching) is distilled from the full model's appearance features via the loss Lapp = sum ||F~_j^l - sg(F_j^l)||_1, which uses the full ReferenceNet as an external teacher; the low-rank residuals are an approximation, not a renamed prediction. The emotion-control claim is evaluated by an independent emotion classifier (Emo-Acc) and against external baselines, not by the paper's own equations. The only self-citation (Yang et al. 2024, ConsistentAvatar) appears in a related-work sentence describing prior person-specific methods; it is not load-bearing for any derivation. The appendix promises an analysis of actor-specific motion bias, but its absence is a missing-evidence/correctness concern, not circularity. The domain gap between proxy renderings and the real frames used to train E_m/E_k is a generalization risk, not a circular reduction.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The system relies on learned low-rank adapters and hand-chosen hyperparameters. No new physical entities are introduced. The key domain assumptions are the generalization of a single-identity proxy and the low-rank structure of appearance feature evolution across denoising steps, both of which are only partially validated in the provided text.

free parameters (3)
  • Low-rank appearance residual adapters (A_j^l, B_j^l) = learned by minimizing L_app
    These matrices approximate feature changes across denoising steps; they are fitted to the teacher's full features via distillation.
  • Number of distilled denoising steps K = K=4 in experiments
    Hand-chosen schedule for speed-quality tradeoff.
  • Appearance distillation loss weight lambda_app = not specified in main text
    Balances L_step and L_app; value deferred to appendix.
axioms (4)
  • standard math 3D Gaussian splatting is an appropriate differentiable renderer for the proxy avatar.
    Used in Eq. 1-3 to define the proxy representation and rendering.
  • domain assumption The difference between appearance features across denoising steps is small and structured, so F_j can be approximated by F_1 + A_j*B_j.
    This is the core assumption of low-rank appearance reuse in Section 'Low-rank Appearance Reuse'.
  • domain assumption A single identity's emotional motion generalizes to arbitrary target portraits.
    The proxy avatar is trained on one MEAD subject; the paper promises an analysis of actor-specific bias in the appendix, which is not present.
  • domain assumption Portrait structure and motion are mainly established in the early denoising steps.
    Reported in the appendix (not included) and used to justify few-step distillation.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Proxy Avatar Meets Low-Rank Caching: Real-Time One-Shot Emotion-Controllable Portrait Animation." pith.science (2026). https://pith.science/paper/P3SUFYTL

@misc{pith2026260801978,
  author       = {Pith},
  title        = {Pith review of: Proxy Avatar Meets Low-Rank Caching: Real-Time One-Shot Emotion-Controllable Portrait Animation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/P3SUFYTL}},
  note         = {Machine review of arXiv:2608.01978}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Audio-driven portrait animation has advanced rapidly with diffusion-based generative models, yet real-time one-shot generation with expressive emotion control remains challenging. Existing methods often suffer from insufficient emotion-aware motion priors and expensive appearance computation during multi-step denoising. To address these issues, we propose Proxy Avatar Meets Low-Rank Caching, a cascaded framework for real-time one-shot emotion-controllable portrait animation. Instead of directly generating the target portrait from audio, our method uses a Gaussian-based emotion proxy avatar as a reusable motion generator, which is trained once on a single identity to produce expressive driving videos from audio and emotion labels. Since the proxy avatar only provides motion rather than target appearance or geometry, a large-scale one-shot retargeting model further extracts identity-independent motion from the proxy performance and adapts it to arbitrary target portraits. To improve inference efficiency, we introduce zero-shot appearance reuse with low-rank caching, which caches reference appearance features at the initial denoising step and models subsequent feature variations using lightweight low-rank adapters. Extensive experiments demonstrate that our method achieves stronger emotional expressiveness, better identity-preserving animation, and substantially reduced inference cost, enabling real-time one-shot portrait animation.

Figures

Figures reproduced from arXiv: 2608.01978 by Haijie Yang, Hao Tang, Hongliang Zhang, Jian Bi, Jianjun Qian, Jian Yang, Jindi Bao, Yixuan Dong, Zhenyu Zhang.

Figure 1
Figure 1. Figure 1: Given a source portrait, an audio clip, and an emo [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview. A 3D Gaussian-based emotion proxy avatar first produces controllable emotional driving videos from audio and emotion conditions. The one-shot portrait motion driving module then transfers identity-independent proxy motion to arbitrary target portraits. Finally, the distillation acceleration module reduces redundant diffusion computation for efficient real-time portrait animation. realism and expr… view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative comparison on one-shot cross-identity driving. Given a single reference portrait and driving audio from a different identity, our method produces more expres￾sive emotional facial motions while better preserving the target identity. Quantitative Results. Tab. 1 reports the quantitative com￾parison under the self-reconstruction setting. Our method achieves the best performance on most metrics (e… view at source ↗
Figure 5
Figure 5. Figure 5: Qualitative ablation results. The left part shows [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

110 extracted references · 13 linked inside Pith

  1. [1]

    Communication, Simulation, and Intelligent Agents: Implications of Personal Intelligent Machines for Medical Education

    Clancey, William J. Communication, Simulation, and Intelligent Agents: Implications of Personal Intelligent Machines for Medical Education. Proceedings of the Eighth International Joint Conference on Artificial Intelligence (IJCAI-83)

  2. [2]

    Classification Problem Solving

    Clancey, William J. Classification Problem Solving. Proceedings of the Fourth National Conference on Artificial Intelligence

  3. [3]

    , title =

    Robinson, Arthur L. , title =. 1980 , doi =. https://science.sciencemag.org/content/208/4447/1019.full.pdf , journal =

    1980

  4. [4]

    New Ways to Make Microcircuits Smaller---Duplicate Entry

    Robinson, Arthur L. New Ways to Make Microcircuits Smaller---Duplicate Entry. Science

  5. [5]

    Clancey and Glenn Rennels , abstract =

    Diane Warner Hasling and William J. Clancey and Glenn Rennels , abstract =. Strategic explanations for a diagnostic consultation system , journal =. 1984 , issn =. doi:https://doi.org/10.1016/S0020-7373(84)80003-6 , url =

    doi:10.1016/s0020-7373(84)80003-6 1984

  6. [6]

    and Rennels, Glenn R

    Hasling, Diane Warner and Clancey, William J. and Rennels, Glenn R. and Test, Thomas. Strategic Explanations in Consultation---Duplicate. The International Journal of Man-Machine Studies

  7. [7]

    Poligon: A System for Parallel Problem Solving

    Rice, James. Poligon: A System for Parallel Problem Solving

  8. [8]

    Transfer of Rule-Based Expertise through a Tutorial Dialogue

    Clancey, William J. Transfer of Rule-Based Expertise through a Tutorial Dialogue

  9. [9]

    The Engineering of Qualitative Models

    Clancey, William J. The Engineering of Qualitative Models

  10. [10]

    2023 , eprint=

    Attention Is All You Need , author=. 2023 , eprint=

    2023

  11. [11]

    Pluto: The 'Other' Red Planet

    NASA. Pluto: The 'Other' Red Planet

  12. [12]

    Deep Audio-Visual Speech Recognition , volume=

    Afouras, Triantafyllos and Chung, Joon Son and Senior, Andrew and Vinyals, Oriol and Zisserman, Andrew , year=. Deep Audio-Visual Speech Recognition , volume=. IEEE Transactions on Pattern Analysis and Machine Intelligence , publisher=. doi:10.1109/tpami.2018.2889052 , number=

    arXiv 2018

  13. [13]

    2022 , eprint=

    Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction , author=. 2022 , eprint=

    2022

  14. [14]

    2023 , eprint=

    Jointly Learning Visual and Auditory Speech Representations from Raw Data , author=. 2023 , eprint=

    2023

  15. [15]

    Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labels , url=

    Ma, Pingchuan and Haliassos, Alexandros and Fernandez-Lopez, Adriana and Chen, Honglie and Petridis, Stavros and Pantic, Maja , year=. Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labels , url=. doi:10.1109/icassp49357.2023.10096889 , booktitle=

    arXiv 2023

  16. [16]

    2021 , eprint=

    End-to-end Audio-visual Speech Recognition with Conformers , author=. 2021 , eprint=

    2021

  17. [17]

    2022 , eprint=

    Visual Context-driven Audio Feature Enhancement for Robust End-to-End Audio-Visual Speech Recognition , author=. 2022 , eprint=

    2022

  18. [18]

    2022 , eprint=

    u-HuBERT: Unified Mixed-Modal Speech Pretraining And Zero-Shot Transfer to Unlabeled Modality , author=. 2022 , eprint=

    2022

  19. [19]

    2024 , eprint=

    Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputs , author=. 2024 , eprint=

    2024

  20. [20]

    2025 , eprint=

    Large Language Models are Strong Audio-Visual Speech Recognition Learners , author=. 2025 , eprint=

    2025

  21. [21]

    2025 , eprint=

    MMS-LLaMA: Efficient LLM-based Audio-Visual Speech Recognition with Minimal Multimodal Speech Tokens , author=. 2025 , eprint=

    2025

  22. [22]

    2024 , eprint=

    Matryoshka Multimodal Models , author=. 2024 , eprint=

    2024

  23. [23]

    2024 , eprint=

    Matryoshka Query Transformer for Large Vision-Language Models , author=. 2024 , eprint=

    2024

  24. [24]

    2025 , eprint=

    Adaptive Audio-Visual Speech Recognition via Matryoshka-Based Multimodal LLMs , author=. 2025 , eprint=

    2025

  25. [25]

    2025 , eprint=

    MoME: Mixture of Matryoshka Experts for Audio-Visual Speech Recognition , author=. 2025 , eprint=

    2025

  26. [26]

    HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units , year=

    Hsu, Wei-Ning and Bolte, Benjamin and Tsai, Yao-Hung Hubert and Lakhotia, Kushal and Salakhutdinov, Ruslan and Mohamed, Abdelrahman , journal=. HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units , year=

  27. [27]

    2015 , eprint=

    Deep Speech 2: End-to-End Speech Recognition in English and Mandarin , author=. 2015 , eprint=

    2015

  28. [28]

    2017 , eprint=

    Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learning , author=. 2017 , eprint=

    2017

  29. [29]

    NOISEX-92: A database and an experiment to study the effect of additive noise on speech recognition systems , journal =

    Assessment for automatic speech recognition: II. NOISEX-92: A database and an experiment to study the effect of additive noise on speech recognition systems , journal =. 1993 , issn =

    1993

  30. [30]

    2017 , eprint=

    End-To-End Visual Speech Recognition With LSTMs , author=. 2017 , eprint=

    2017

  31. [31]

    2020 , eprint=

    Lipreading using Temporal Convolutional Networks , author=. 2020 , eprint=

    2020

  32. [32]

    2023 , eprint=

    OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment , author=. 2023 , eprint=

    2023

  33. [33]

    Deep multimodal learning for Audio-Visual Speech Recognition , year=

    Mroueh, Youssef and Marcheret, Etienne and Goel, Vaibhava , booktitle=. Deep multimodal learning for Audio-Visual Speech Recognition , year=

  34. [34]

    Audio-visual deep learning for noise robust speech recognition , year=

    Huang, Jing and Kingsbury, Brian , booktitle=. Audio-visual deep learning for noise robust speech recognition , year=

  35. [35]

    and Kaiser,

    Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N. and Kaiser,. Attention is all you need , year =

  36. [36]

    2018 , eprint=

    Audio-Visual Speech Recognition With A Hybrid CTC/Attention Architecture , author=. 2018 , eprint=

    2018

  37. [37]

    2024 , eprint=

    AV-data2vec: Self-supervised Learning of Audio-Visual Speech Representations with Contextualized Target Representations , author=. 2024 , eprint=

    2024

  38. [38]

    ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , year=

    Connecting Speech Encoder and Large Language Model for ASR , author=. ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , year=

    2024

  39. [39]

    2024 , eprint=

    LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPT , author=. 2024 , eprint=

    2024

  40. [40]

    2023 , eprint=

    SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities , author=. 2023 , eprint=

    2023

  41. [41]

    2023 , eprint=

    Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models , author=. 2023 , eprint=

    2023

  42. [42]

    arXiv preprint arXiv:2302.13971 , year=

    LLaMA: Open and Efficient Foundation Language Models , author=. arXiv preprint arXiv:2302.13971 , year=

    Pith/arXiv arXiv

  43. [43]

    Radhakrishnan, Sreyan and Yang, Chao-Han Huck and Khan, S. A. and Kumar, R. and Kiani, N. A. and Gomez-Cabrero, D. and Tegner, J. N. , booktitle=. Whispering

  44. [44]

    Advances in Neural Information Processing Systems (NeurIPS) , year=

    Hyporalise: An Open Baseline for Generative Speech Recognition with Large Language Models , author=. Advances in Neural Information Processing Systems (NeurIPS) , year=

  45. [45]

    Crossmodal

    Li, Yizhi and Chen, Peng and Bell, Peter and Lai, Catherine , booktitle=. Crossmodal

  46. [46]

    arXiv preprint arXiv:2309.16609 , year=

    Qwen Technical Report , author=. arXiv preprint arXiv:2309.16609 , year=

    Pith/arXiv arXiv

  47. [47]

    and Tang, C

    Yu, W. and Tang, C. and Sun, G. and Chen, X. and Tan, T. and Li, W. and Lu, L. and Ma, Z. and Zhang, C. , booktitle=. Connecting Speech Encoder and Large Language Model for

  48. [48]

    Li, Junnan and Li, Dongxu and Savarese, Steven and Hoi, Steven , booktitle=

  49. [49]

    Computer Vision -

    Kaisiyuan Wang and Qianyi Wu and Linsen Song and Zhuoqian Yang and Wayne Wu and Chen Qian and Ran He and Yu Qiao and Chen Change Loy , editor =. Computer Vision -. 2020 , timestamp =

    2020

  50. [50]

    arXiv preprint arXiv:1809.00496 , year=

    LRS3-TED: a large-scale dataset for visual speech recognition , author=. arXiv preprint arXiv:1809.00496 , year=

    Pith/arXiv arXiv

  51. [51]

    arXiv preprint arXiv:1806.05622 , year=

    VoxCeleb2: Deep speaker recognition , author=. arXiv preprint arXiv:1806.05622 , year=

    Pith/arXiv arXiv

  52. [52]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    Retinaface: Single-shot multi-level face localisation in the wild , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  53. [53]

    arXiv preprint arXiv:1412.6980 , year =

    Adam: A Method for Stochastic Optimization , author =. arXiv preprint arXiv:1412.6980 , year =

    Pith/arXiv arXiv

  54. [54]

    2024 , eprint=

    BRAVEn: Improving Self-Supervised Pre-training for Visual and Auditory Speech Recognition , author=. 2024 , eprint=

    2024

  55. [55]

    2024 , eprint=

    Multilingual Audio-Visual Speech Recognition with Hybrid CTC/RNN-T Fast Conformer , author=. 2024 , eprint=

    2024

  56. [56]

    Whisper-Flamingo: Integrating Visual Features into Whisper for Audio-Visual Speech Recognition and Translation , author=

  57. [57]

    2022 , eprint=

    Transformer-Based Video Front-Ends for Audio-Visual Speech Recognition for Single and Multi-Person Video , author=. 2022 , eprint=

    2022

  58. [58]

    2024 , eprint=

    Learning Video Temporal Dynamics with Cross-Modal Attention for Robust Audio-Visual Speech Recognition , author=. 2024 , eprint=

    2024

  59. [59]

    arXiv preprint arXiv:2106.09685 , year=

    Lora: Low-rank adaptation of large language models , author=. arXiv preprint arXiv:2106.09685 , year=

    Pith/arXiv arXiv

  60. [60]

    International Conference on Learning Representations (ICLR) , year=

    Matryoshka Representation Learning , author=. International Conference on Learning Representations (ICLR) , year=

  61. [61]

    Neural Computation , volume =

    Adaptive mixtures of local experts , author =. Neural Computation , volume =. 1991 , publisher =

    1991

  62. [62]

    Physiological reviews , volume=

    The brain basis of language processing: from structure to function , author=. Physiological reviews , volume=. 2011 , publisher=

    2011

  63. [63]

    2008 , publisher=

    Intonational phonology , author=. 2008 , publisher=

    2008

  64. [64]

    Proceedings of the 14th python in science conference , volume =

    librosa: Audio and music signal analysis in python , author =. Proceedings of the 14th python in science conference , volume =. 2015 , pages =

    2015

  65. [65]

    arXiv preprint arXiv:2305.14314 , year=

    QLoRA: Efficient Finetuning of Quantized LLMs , author=. arXiv preprint arXiv:2305.14314 , year=

    Pith/arXiv arXiv

  66. [66]

    Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence , year =

    Multi-attention recurrent network for human communication comprehension , author =. Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence , year =

  67. [67]

    International conference on machine learning , pages=

    Robust speech recognition via large-scale weak supervision , author=. International conference on machine learning , pages=. 2023 , organization=

    2023

  68. [68]

    2024 , eprint=

    Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animation , author=. 2024 , eprint=

    2024

  69. [69]

    2023 , eprint=

    VividTalk: One-Shot Audio-Driven Talking Head Generation Based on 3D Hybrid Prior , author=. 2023 , eprint=

    2023

  70. [70]

    2017 , issue_date =

    Karras, Tero and Aila, Timo and Laine, Samuli and Herva, Antti and Lehtinen, Jaakko , title =. 2017 , issue_date =. doi:10.1145/3072959.3073658 , journal =

    arXiv 2017

  71. [71]

    ACM Trans

    3D Gaussian splatting for real-time radiance field rendering , author=. ACM Trans. Graph. , volume=

  72. [72]

    ACM SIGGRAPH 2024 Conference Papers , pages=

    3d gaussian blendshapes for head avatar animation , author=. ACM SIGGRAPH 2024 Conference Papers , pages=

    2024

  73. [73]

    2023 , eprint=

    EmoTalk: Speech-Driven Emotional Disentanglement for 3D Face Animation , author=. 2023 , eprint=

    2023

  74. [74]

    Proceedings of the 32nd ACM International Conference on Multimedia , pages=

    Consistentavatar: Learning to diffuse fully consistent talking head avatar with temporal guidance , author=. Proceedings of the 32nd ACM International Conference on Multimedia , pages=

  75. [75]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    Audio-driven emotional video portraits , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  76. [76]

    ACM SIGGRAPH 2022 conference proceedings , pages=

    Eamm: One-shot emotional talking face via audio-based emotion-aware motion model , author=. ACM SIGGRAPH 2022 conference proceedings , pages=

    2022

  77. [77]

    Proceedings of the AAAI conference on artificial intelligence , volume=

    Styletalk: One-shot talking head generation with controllable speaking styles , author=. Proceedings of the AAAI conference on artificial intelligence , volume=

  78. [78]

    Proceedings of the ieee/cvf international conference on computer vision , pages=

    Space: Speech-driven portrait animation with controllable expression , author=. Proceedings of the ieee/cvf international conference on computer vision , pages=

  79. [79]

    IEEE Transactions on Multimedia , year=

    Emospeaker: One-shot fine-grained emotion-controlled talking face generation , author=. IEEE Transactions on Multimedia , year=

  80. [80]

    Advances in Neural Information Processing Systems , volume=

    Vasa-1: Lifelike audio-driven talking faces generated in real time , author=. Advances in Neural Information Processing Systems , volume=

Showing first 80 references.

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.