Pith. sign in

REVIEW 3 major objections 6 minor 55 references

MemoryTalker: Personalized Speech-Driven 3D Facial Animation via Audio-Guided Stylization

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read MemoryTalker claims a speaker's mouth style — opening, pouting, articulation — can be read from audio and personalize 3D animation for unseen voices, with no labels or reference meshes at inference.

desk verdict Competent and clean two-stage design for audio-only 3D facial animation, but the headline claim of arbitrary new-voice generalization is untested. read the letter →

arxiv 2507.20562 v2 pith:XLSB2MSS submitted 2025-07-28 cs.CV cs.AI

Hyung Kyu Kim , Sangmin Lee , Hak Gu Kim This is my paper · ORCID
classification cs.CVcs.AI
keywords speech-driven3Dfacialanimationspeakingstyleaudio-guidedstylizationmotionmemorynetworkkey-valuestyle-contentdisentanglementpersonalizedVOCASET
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Speech-driven 3D facial animation must do two things at once: move the lips in sync with the words, and move them the way that particular speaker moves them. Prior systems either need a speaker-identity label, which cannot describe a voice never seen in training, or a reference 3D facial mesh at inference, which is impractical for real applications. This paper claims both needs can be met from the audio signal alone. Its two-stage scheme first stores generic, speaker-neutral lip motion in a learnable motion memory keyed by a speech-recognition representation of the words, then rescales every memory slot by a 'speaking style' vector extracted from a mel-spectrogram of the same audio and trained with a triplet loss to pull same-speaker styles together. If the claim holds, an arbitrary new voice is enough to drive a personalized avatar in real time — about 120 frames per second with 94M parameters — and the paper reports the lowest error on every metric on both VOCASET and BIWI, including lip-region error $0.293$ vs $0.375$ for the previous best method.

What carries the argument

The load-bearing object is the key-value motion memory $\mathbf{M}_m \in \mathbb{R}^{n\times c}$, with 32 slots, which plays two roles across the two training stages. In stage 1 (Memorizing) it stores and retrieves general facial motion: a motion encoder maps vertex displacements into query features, while text representations from a pre-trained ASR encoder form a softmax key-address vector, and a KL-divergence loss aligns the two addressing distributions so that the same word retrieves canonical, speaker-neutral lip motion. In stage 2 (Animating) the style feature $f_s$ from a mel-spectrogram encoder converts each slot into a scaled version, $\tilde{\mathbf{M}}_m = \{\tilde{w}^i_s \cdot s^i_m\}_{i=1}^n$ with $\tilde{\mathbf{w}}_s = \mathrm{sigmoid}(\psi'_{\to n}(f_s))\cdot\psi_{\to 1}(f_s)$, so recall from the stylized memory — decoded by a Transformer together with the text feature — yields motion whose lip shapes carry that speaker's articulation habits. The mechanism matters because it separates content from style at the memory level itself: the ASR key decides what the mouth does, the style weights decide how strongly, and that separation is what allows generic motion and individual style to be learned in different stages and combined with audio as the only input.

What would settle it

Train on a proper subset of speakers (for example, 10 of VOCASET's 12) and animate the held-out two; if the generated lip motion for the unseen voices is no more accurate than a speaker-neutral baseline, or if their style features collapse onto training clusters in t-SNE rather than forming their own, the audio-only personalization claim is falsified. A cheaper probe on the same point: take one speaker saying the same sentence twice, once over-articulated and once mumbled, and check that the style feature — and the generated mouth-opening amplitude — tracks articulation effort rather than only speaker identity.

Watch

Extended reading notes

Core claim

The central claim is that speaking style is a property of the audio waveform itself — not of an identity slot and not of a reference mesh — and that a network can learn to extract it and inject it into animation. MemoryTalker encodes the words with a pre-trained ASR encoder to address a 32-slot key-value motion memory and recall what the paper calls general motion: the canonical lip shapes that all speakers share for each phoneme. A separate style encoder maps a mel-spectrogram of the same audio to a style feature $f_s$, trained with a triplet loss $\mathcal{L}_{\mathrm{style}}$ so clips of the same speaker cluster together, and that feature rescales the memory slots (via $\tilde{\mathbf{w}}_s = \mathrm{sigmoid}(\psi'_{\to n}(f_s))\cdot\psi_{\to 1}(f_s)$) into a stylized memory $\tilde{\mathbf{M}}_m$ from which personalized motion is recalled and decoded alongside the text feature. The paper reports that this construction outperforms every compared method on every metric on both datasets — on VOCASET, face vertex error $0.506$ vs $0.570$ and lip vertex error $0.293$ vs $0.375$ for the previous best — wins its user study on lip-sync, realism, and speaking style, and runs at about 120 fps with 94M parameters, with audio as the only inference input.

Load-bearing premise

The load-bearing premise, stated in Section 3.2, is that the mel-spectrogram style feature trained with a triplet loss isolates a speaker's articulation style from phonetic content and transfers to voices never heard in training; the figures show training speakers' features separating, but nothing in the paper demonstrates what happens to the style feature — or to the lip motion it produces — for a genuinely unseen voice.

Editorial extensions

If this is right

  • A new voice becomes a complete personalization signal: with audio as the only inference input, any unseen speaker can drive an avatar in their own articulation style, with no enrollment video, identity label, or mesh sequence.
  • The mechanism transfers across mesh topologies: trained on VOCASET's 5,023-vertex FLAME mesh, the same model evaluates directly on BIWI's denser 23,370-vertex mesh and still leads on every metric.
  • The system sits in an interactive range: 7.8 ms per frame and 94M parameters (about 120 fps on a single GPU), well below the cost of the transformer-based animators it is compared with.
  • The two-stage curriculum is itself part of the gain: two-stage memorization-then-stylization training beats end-to-end training (FVE $0.506$ vs $0.510$, LVE $0.293$ vs $0.303$), and ablations show that removing either the memory or the style stage degrades results.
  • Personalization re-uses a shared articulatory basis: key-address visualizations in the supplementary show the same memory slots activating for the same lip shapes across different speakers, so the stylized memory rescales common geometry knowledge rather than memorizing a per-identity template.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because content and style are separated at the memory level, the same recipe — a content key addressing shared slots, a style vector rescaling them — could transfer to adjacent generation tasks such as 2D talking faces, head pose, upper-face expression, or co-speech gesture; the paper itself does not claim this extension.
  • The paper does not run the decisive speaker-generalization ablation: hold entire speakers out of training (not just test sentences) and check whether each unseen voice's style feature still separates in the latent space and whether its animations beat a speaker-neutral baseline on lip error. The t-SNE evidence shows only that training speakers separate, which is necessary but not sufficient for th
  • The triplet style space is anchored to training-set identity, so an adversarial probe — the same sentence spoken over-articulated versus mumbled by the same voice — would reveal whether the encoder tracks articulation effort, which the animation claim needs, or merely voice timbre, which would also separate speakers in t-SNE.
  • If the style feature behaves as claimed, it could serve as a portable style prior beyond animation: given an arbitrary recording, the same $f_s$ could drive style transfer, dubbing, or voice-conversion pipelines that need lip motion matched to a target voice rather than to the source audio.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. MemoryTalker proposes a two-stage framework for speech-driven 3D facial animation in which, at inference, only audio is required. In stage 1, a motion memory stores general facial motion features that are retrieved via text representations from a pretrained ASR encoder, aligned to motion features with an alignment loss. In stage 2, a speaker style encoder maps a mel-spectrogram to a style feature, a triplet loss separates speakers, and the memory slots are rescaled by style-dependent weights so that the recalled motion is personalized. The paper reports quantitative results on VOCASET and BIWI across FVE, LVE, FID, LDTW, and Lip-max, an ablation study, a memory-slot-size study, efficiency measurements, and a user study, claiming state-of-the-art performance without ID labels or mesh inputs at test time.

Significance. If the central claim holds, the practical contribution is clear: removing ID labels and reference meshes at inference makes personalized 3D facial animation more deployable. The two-stage memory design is well specified and the paper includes a reasonable amount of ablative and perceptual evidence. The method is also efficient in parameter count and inference time. However, the paper's headline claim of audio-only personalization for arbitrary unseen voices is not supported by the current evaluation design, which lacks a direct held-out-speaker generalization test and statistical confidence measures. The underlying idea is defensible and the gaps are addressable, so the appropriate path is a major revision.

major comments (3)
  1. [Section 3.2, Eq. (12)-(13), and Table 1] The central claim that MemoryTalker personalizes an arbitrary new voice from audio alone is not established by the reported experiments. The style encoder Es is trained with a triplet loss whose classes are the training speakers, and the quantitative results in Table 1 are averaged over VOCA-Test/BIWI-Test-B without a per-held-out-speaker breakdown. The t-SNE in Fig. 6 and Fig. S8 clusters mostly the training-pool speakers, and Fig. S8 only compares encoding types, so the plots do not show that fs transfers to unseen voices. To support the central claim, please report leave-one-speaker-out experiments in which stage-2 training excludes the test speakers, per-speaker FVE/LVE on the held-out split, and a content-invariance experiment (e.g., same sentence across speakers versus different sentences for the same speaker, with style-feature retrieval or FVE as the metric). Without this evidence, the gains in Table 1 could reflect memorized voice signatures rather than a transferable articulatory style.
  2. [Tables 1 and 6, Section 4.3 and 4.7] The quantitative comparison lacks statistical confidence measures for MemoryTalker itself. Table 1 reports point estimates with no standard deviations or confidence intervals, although Table 2 reports variability for the one-hot baselines. Several reported differences are small (e.g., BIWI FVE 0.901 vs. 0.919 and LDTW 0.107 vs. 0.109), so run-to-run or identity-to-identity variation could alter the conclusions. The user-study percentages in Table 6 are based on a modest number of participants and are not accompanied by any significance test. Please report means with standard deviations or confidence intervals across runs or test identities and provide a paired significance test such as bootstrap or Wilcoxon for the A/B preferences.
  3. [Section 4.4 and Fig. 4 caption] The evaluation protocol for one-hot-based competitors is not aligned with the deployment setting claimed in the paper. The Fig. 4 caption states that the displayed one-hot encoding results use the ID with the lowest error, which gives those methods access to an oracle identity choice unavailable in practice, while MemoryTalker receives only audio. Please clarify whether the Table 1 numbers use this oracle selection and also report the conventional protocol of averaging over all training identities or using a fixed identity. Table S3 provides a useful seen-identity comparison for BIWI, but the VOCASET comparison should be made consistent across methods and clearly described so that the claimed advantage is not an artifact of the comparison protocol.
minor comments (6)
  1. [Section 4.2] When stating that quantitative evaluations were performed on VOCA-Test and BIWI-Test-B, please explicitly state whether the test identities were completely unseen during stage-2 training and give the speaker-level split; this is important for readers to interpret the personalization claim.
  2. [Eq. (15)] The style weight ws contains a scalar term psi_1(fs) that rescales all memory slots; please ablate or visualize this scalar to show that it is not acting as a simple global amplitude gate rather than a per-slot style modulation.
  3. [Table 3] The efficiency comparison reports a single inference-time value for each method; please state the batch size, sequence length, and hardware conditions under which the timings were measured, and report the standard deviation over repeated runs if the measurement is noisy.
  4. [Section 4.7 and Section E.2 of the supplement] The main text says 33 subjects participated, while the supplement says 2 participants were removed after failing qualification questions; please report the final number of participants consistently in the main text and describe how many questions remained per participant after exclusions.
  5. [Figure S1] The memory-slot-size study reports only LVE; since the number of slots is a free parameter of the model, please report FVE, FID, and LDTW for the same sweep so that the choice of 32 slots is justified across all headline metrics.
  6. [Reproducibility] The paper does not mention plans to release code or pretrained models; given the number of architectural details and hyperparameters, a public implementation would materially help reproducibility.

Circularity Check

0 steps flagged · score 2.0 of 10

No load-bearing circularity; the two-stage training is supervised against ground-truth motion, and the minor self-citations are not used to justify the central personalization claim.

full rationale

The derivation chain is a standard supervised pipeline. Stage 1 stores motion features in a key-value memory using reconstruction and KL-alignment losses (Eqs. 3, 7, 9, 10), and stage 2 learns a speaking-style encoder with a triplet loss (Eq. 13) that rescales the frozen memory (Eqs. 14-15); the final motion is decoded against reference vertices under Lmse, Lvel, and Llip (Eqs. 16-17). Every objective is referenced to ground-truth 3D motion or to training speaker identity, and the reported FVE/LVE/LDTW/FID numbers compare generated vertices to test-set reference vertices, so the central result is externally falsifiable. No fitted parameter is renamed as a prediction: the style feature fs = Es(phi_{a2m}(a)) (Eq. 12) is an encoder output at inference, not a value fitted to the test target, and the stylized memory is evaluated against reference motion rather than defining the metric. There are self-citations ([20], [23], [24]) but they are related-work and architecture context only; no uniqueness theorem or prior-work ansatz is imported to make the personalization step forced. The genuine weakness is that unseen-voice generalization is asserted but not demonstrated: Section 4.2 says evaluations were performed on VOCA-Test and BIWI-Test-B by averaging identities, and the t-SNE evidence (Fig. 6, Fig. S8) shows separation of speakers in the training/evaluation pool rather than transfer to a genuinely new voice. That is a missing-support/empirical-validity concern, not a circularity, so it does not raise the circularity score beyond the minor-self-citation level.

Assumptions & free parameters 5 free parameters · 4 assumptions · 2 invented entities

The central claim rests on a small number of free hyperparameters, most of which are not publicly specified (kappa, triplet margin), and on assumptions about the separability of style and content in speech. The memory bank and stylized memory are architectural entities whose effectiveness is only demonstrated in the paper's own evaluation.

free parameters (5)
  • Memory slot count n = 32
    Selected after a validation sweep over 16/24/32/48/64 slots using LVE (Supplementary B.1); the slot count affects the capacity of the motion memory.
  • Loss weight lambda1 = 0.01
    Weight for Lmem + Lalign in Eq. 11; chosen by the authors, no sensitivity analysis.
  • Loss weight lambda2 = 0.01
    Weight for Llip + Lstyle in Eq. 17; chosen by the authors, no sensitivity analysis.
  • Triplet margin l = not specified
    Margin in Eq. 13; its value is not stated in the paper or supplement.
  • Cosine scaling factor kappa = not specified
    Scaling factor in the memory attention in Eq. 1; value is not stated.
assumptions (4)
  • domain assumption HuBERT ASR text representations are phonetically aligned with articulation and are speaker-normal enough to retrieve general motion.
    Section 3.1, Eq. 4: pre-trained HuBERT is used to map audio to text-like features that act as keys into the motion memory. The method assumes these features cluster by phoneme rather than by speaker.
  • domain assumption The mel-spectrogram of speech contains a speaker-specific style signal that is separable from phonetic content.
    Section 3.2, Eq. 12: the style encoder Es takes a mel-spectrogram and produces fs. The whole second stage depends on this separability.
  • domain assumption A triplet loss over training speaker IDs produces style features that generalize to unseen speakers.
    Section 3.2, Eq. 13: Lstyle is defined using speakers in the training set; the model then applies fs at inference to new voices. Generalization is assumed, not proven.
  • domain assumption Key-value memory slots can store and retrieve articulatory motion from text queries.
    Section 3.1, Eqs. 1-6: the design assumes that an n-slot memory with soft attention can faithfully store and recall facial motion features.
invented entities (2)
  • Motion memory bank M_m
    purpose: A set of n learnable slots that store generic facial motion features, accessed by text and motion queries.
    The memory bank is the core architectural invention. It has no independent evidence outside the paper's benchmark experiments; its success is measured only indirectly through FVE/LVE.
  • Stylized motion memory M_m_tilde
    purpose: The same motion memory after a speaker-specific rescaling of slots, driven by the audio style feature.
    Eq. 14 defines M_m_tilde. It is a new entity that stores personalized motion; no independent falsifiable handle is given beyond the paper's own evaluations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MemoryTalker: Personalized Speech-Driven 3D Facial Animation via Audio-Guided Stylization." pith.science (2026). https://pith.science/paper/XLSB2MSS

@misc{pith2026250720562,
  author       = {Pith},
  title        = {Pith review of: MemoryTalker: Personalized Speech-Driven 3D Facial Animation via Audio-Guided Stylization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XLSB2MSS}},
  note         = {Machine review of arXiv:2507.20562}
}
read the original abstract

Speech-driven 3D facial animation aims to synthesize realistic facial motion sequences from given audio, matching the speaker's speaking style. However, previous works often require priors such as class labels of a speaker or additional 3D facial meshes at inference, which makes them fail to reflect the speaking style and limits their practical use. To address these issues, we propose MemoryTalker which enables realistic and accurate 3D facial motion synthesis by reflecting speaking style only with audio input to maximize usability in applications. Our framework consists of two training stages: 1-stage is storing and retrieving general motion (i.e., Memorizing), and 2-stage is to perform the personalized facial motion synthesis (i.e., Animating) with the motion memory stylized by the audio-driven speaking style feature. In this second stage, our model learns about which facial motion types should be emphasized for a particular piece of audio. As a result, our MemoryTalker can generate a reliable personalized facial animation without additional prior information. With quantitative and qualitative evaluations, as well as user study, we show the effectiveness of our model and its performance enhancement for personalized facial animation over state-of-the-art methods.

Figures

Figures reproduced from arXiv: 2507.20562 by the authors.

Figure 1
Figure 1. The intuition of MemoryTalker for personalized speech￾driven 3D facial animation, Memorizing and Animating. Memoriz￾ing: Storing and retrieving facial motion. Animating: Synthesizing the personalized 3D facial motion with the stylized motion memory. Our MemoryTalker can accurately produce the personalized 3D facial motion for different speakers using only audio input. The core challenge lies in developing algorithms… view at source ↗
Figure 2
Figure 2. Explored approaches for personalized speech-driven 3D facial animation. (a) In the one-hot encoding approaches, while the [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Illustration of the proposed MemoryTalker model for personalized 3D facial animation. (a) Learning to store facial motion feature in the facial motion memory and align motion features with text features. (b) Learning to disentangle unique speaking style of each speaker from audio and stylizing facial motion memory. 2. Related Work 2.1. Speech-Driven 3D Facial Animation Earlier methods [1, 7, 9] for 3D facial animati… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Visual comparisons with state-of-the art methods on (a) VOCASET and (b) BIWI. Note that the second and fourth rows represent [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Qualitative results from “w/o 2-stage” and “w/ 2-stage”. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: The t-SNE visualization of the recalled motion features [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

55 extracted references · 53 canonical work pages

  1. [1]

    Verma, N

    A. Verma, N. Rajput, and L.V . Subramaniam. Using viseme based acoustic models for speech driven lip synthesis. In 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings. (ICASSP ’03)., pages V–720, 2003. 3

    work page 2003

  2. [2]

    Facetalk: Audio-driven motion diffusion for neural parametric head models

    Shivangi Aneja, Justus Thies, Angela Dai, and Matthias Nießner. Facetalk: Audio-driven motion diffusion for neural parametric head models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21263–21273, 2024. 1

    work page 2024

  3. [3]

    Personalized Speech-driven Expressive 3D Facial Animation Synthesis with Style Control

    Elif Bozkurt. Personalized speech-driven expressive 3d facial animation synthesis with style control. arXiv preprint arXiv:2310.17011, 2023. 7

    work page Pith review arXiv 2023

  4. [4]

    Memory matching networks for one-shot image recognition

    Qi Cai, Yingwei Pan, Ting Yao, Chenggang Yan, and Tao Mei. Memory matching networks for one-shot image recognition. In Proceedings of the IEEE con- ference on computer vision and pattern recognition , pages 4080–4088, 2018. 4

    work page 2018

  5. [5]

    Avface: Towards detailed audio-visual 4d face reconstruction

    Aggelina Chatziagapi and Dimitris Samaras. Avface: Towards detailed audio-visual 4d face reconstruction. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 16878– 16889, 2023. 1

    work page 2023

  6. [6]

    Capture, learning, and synthesis of 3D speaking styles

    Daniel Cudeiro, Timo Bolkart, Cassidy Laidlaw, Anurag Ranjan, and Michael Black. Capture, learning, and synthesis of 3D speaking styles. Computer Vision and Pattern Recognition (CVPR), pages 10101–10111,

    work page

  7. [7]

    Facial animation based on context-dependent visemes

    Jos´e Mario De Martino, L´eo Pini Magalh˜aes, and F´abio Violaro. Facial animation based on context-dependent visemes. Computers & Graphics, 30(6):971–980, 2006. 3

    work page 2006

  8. [8]

    Disentanglement of emotional style and speaker identity for expressive voice conversion

    Zongyang Du, Berrak Sisman, Kun Zhou, and Haizhou Li. Disentanglement of emotional style and speaker identity for expressive voice conversion. InInterspeech 2022, pages 2603–2607, 2022. 6

    work page 2022

Show all 55 references
  1. [9]

    Jali: an animator-centric viseme model for ex- pressive lip synchronization

    Pif Edwards, Chris Landreth, Eugene Fiume, and Karan Singh. Jali: an animator-centric viseme model for ex- pressive lip synchronization. ACM Transactions on graphics (TOG), 35(4):1–11, 2016. 3

    2016

  2. [10]

    Enhancing speech-driven 3d facial an- imation with audio-visual guidance from lip reading expert

    Han EunGi, Oh Hyun-Bin, Kim Sung-Bin, Corentin Nivelet Etcheberry, Suekyeong Nam, Janghoon Ju, and Tae-Hyun Oh. Enhancing speech-driven 3d facial an- imation with audio-visual guidance from lip reading expert. In Interspeech 2024, pages 2940–2944, 2024. 1

    2024

  3. [11]

    Unitalker: Scaling up audio-driven 3d facial animation through a unified model

    Xiangyu Fan, Jiaqi Li, Zhiqian Lin, Weiye Xiao, and Lei Yang. Unitalker: Scaling up audio-driven 3d facial animation through a unified model. In European Con- ference on Computer Vision, pages 204–221. Springer,

  4. [12]

    Faceformer: Speech-driven 3d facial animation with transformers

    Yingruo Fan, Zhaojiang Lin, Jun Saito, Wenping Wang, and Taku Komura. Faceformer: Speech-driven 3d facial animation with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 1, 3, 5, 6, 7, 8, 9

    2022

  5. [13]

    A 3-d audio-visual corpus of affective communication

    Gabriele Fanelli, Juergen Gall, Harald Romsdorfer, Thibaut Weise, and Luc Van Gool. A 3-d audio-visual corpus of affective communication. IEEE Transactions on Multimedia, 12(6):591–598, 2010. 1, 6

    2010

  6. [14]

    Stmtrack: Template-free visual tracking with space-time memory networks

    Zhihong Fu, Qingjie Liu, Zehua Fu, and Yunhong Wang. Stmtrack: Template-free visual tracking with space-time memory networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13774–13783, 2021. 4

    2021

  7. [15]

    Memorizing normality to detect anomaly: Memory-augmented deep autoencoder for un- supervised anomaly detection

    Dong Gong, Lingqiao Liu, Vuong Le, Budhaditya Saha, Moussa Reda Mansour, Svetha Venkatesh, and An- ton van den Hengel. Memorizing normality to detect anomaly: Memory-augmented deep autoencoder for un- supervised anomaly detection. In IEEE International Conference on Computer V...

    2019

  8. [16]

    Speech4mesh: Speech-assisted monocular 3d facial reconstruction for speech-driven 3d facial ani- mation

    Shan He, Haonan He, Shuo Yang, Xiaoyan Wu, Pengcheng Xia, Bing Yin, Cong Liu, Lirong Dai, and Chang Xu. Speech4mesh: Speech-assisted monocular 3d facial reconstruction for speech-driven 3d facial ani- mation. In Proceedings of the IEEE/CVF International Conference on Computer ...

  9. [17]

    Gans trained by a two time-scale update rule converge to a local nash equilibrium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information pro- cessing systems, 30, 2017. 7

    2017

  10. [18]

    Hubert: Self-supervised speech representation learning by masked prediction of hidden units

    Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Ab- delrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2...

    2021

  11. [19]

    Mimic: Speaking style disentanglement for speech- driven 3d facial animation

    Hui Fu, Zeqing Wang, Ke Gong, Keze Wang, Tianshui Chen, Haojie Li, Haifeng Zeng, and Wenxiong Kang. Mimic: Speaking style disentanglement for speech- driven 3d facial animation. In The 38th Annual AAAI Conference on Artificial Intelligence (AAAI), 2024. 2, 3, 1, 5, 6

    2024

  12. [20]

    An- alyzing visible articulatory movements in speech pro- duction for speech-driven 3d facial animation

    Hyung Kyu Kim, Sangmin Lee, and Hak Gu Kim. An- alyzing visible articulatory movements in speech pro- duction for speech-driven 3d facial animation. In 2024 IEEE International Conference on Image Processing (ICIP), pages 3575–3579. IEEE, 2024. 1

    2024

  13. [21]

    Multi-modality associative bridging through mem- ory: Speech sound recollected from face video

    Minsu Kim, Joanna Hong, Se Jin Park, and Yong Man Ro. Multi-modality associative bridging through mem- ory: Speech sound recollected from face video. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pages 296–306, 2021. 4

    2021

  14. [22]

    Dis- tinguishing homophenes using multi-head visual-audio memory for lip reading

    Minsu Kim, Jeong Hun Yeo, and Yong Man Ro. Dis- tinguishing homophenes using multi-head visual-audio memory for lip reading. In Proceedings of the AAAI conference on artificial intelligence, pages 1174–1182,

  15. [23]

    Video prediction recalling long-term motion context via memory alignment learn- ing

    Sangmin Lee, Hak Gu Kim, Dae Hwi Choi, Hyung-Il Kim, and Yong Man Ro. Video prediction recalling long-term motion context via memory alignment learn- ing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021. 4

    2021

  16. [24]

    Weakly paired associative learning for sound and im- age representations via bimodal associative memory

    Sangmin Lee, Hyung-Il Kim, and Yong Man Ro. Weakly paired associative learning for sound and im- age representations via bimodal associative memory. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 10534– 10543, 2022. 4

    2022

  17. [25]

    Learning a model of facial shape and expression from 4d scans

    Tianye Li, Timo Bolkart, Michael J Black, Hao Li, and Javier Romero. Learning a model of facial shape and expression from 4d scans. ACM Trans. Graph., 36(6): 194–1, 2017. 6

    2017

  18. [26]

    Emage: To- wards unified holistic co-speech gesture generation via expressive masked audio gesture modeling

    Haiyang Liu, Zihao Zhu, Giorgio Becherini, Yichen Peng, Mingyang Su, You Zhou, Xuefei Zhe, Naoya Iwamoto, Bo Zheng, and Michael J Black. Emage: To- wards unified holistic co-speech gesture generation via expressive masked audio gesture modeling. InProceed- ings of the IEEE/CVF...

    2024

  19. [27]

    Geometry-guided dense perspective network for speech-driven facial animation

    Jingying Liu, Binyuan Hui, Kun Li, Yunke Liu, Yu- Kun Lai, Yuxiang Zhang, Yebin Liu, and Jingyu Yang. Geometry-guided dense perspective network for speech-driven facial animation. IEEE Transactions on Visualization and Computer Graphics, 28(12):4873– 4886, 2021. 1

    2021

  20. [28]

    Key-value memory networks for directly reading doc- uments

    Alexander Miller, Adam Fisch, Jesse Dodge, Amir- Hossein Karimi, Antoine Bordes, and Jason Weston. Key-value memory networks for directly reading doc- uments. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1400–1409, Austin, Te...

    2016

  21. [29]

    Network anomaly detection using memory-augmented deep autoencoder

    Byeongjun Min, Jihoon Yoo, Sangsoo Kim, Dongil Shin, and Dongkyoo Shin. Network anomaly detection using memory-augmented deep autoencoder. IEEE Access, 9:104695–104706, 2021. 4

    2021

  22. [30]

    Graph- level anomaly detection via hierarchical memory net- works

    Chaoxi Niu, Guansong Pang, and Ling Chen. Graph- level anomaly detection via hierarchical memory net- works. In Machine Learning and Knowledge Discovery in Databases: Research Track: European Conference, ECML PKDD 2023, Turin, Italy, September 18–22, 2023, Proceedings, Part I,...

    2023

  23. [31]

    Scantalk: 3d talking heads from unregistered scans

    Federico Nocentini, Thomas Besnier, Claudio Ferrari, Sylvain Arguillere, Stefano Berretti, and Mohamed Daoudi. Scantalk: 3d talking heads from unregistered scans. In European Conference on Computer Vision, pages 19–36. Springer, 2024. 1, 6, 8

    2024

  24. [32]

    Synctalkface: Talking face gener- ation with precise lip-syncing via audio-lip memory

    Se Jin Park, Minsu Kim, Joanna Hong, Jeongsoo Choi, and Yong Man Ro. Synctalkface: Talking face gener- ation with precise lip-syncing via audio-lip memory. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 2062–2070, 2022. 4

    2022

  25. [33]

    Self- talk: A self-supervised commutative training diagram to comprehend 3d talking faces

    Ziqiao Peng, Yihao Luo, Yue Shi, Hao Xu, Xiangyu Zhu, Hongyan Liu, Jun He, and Zhaoxin Fan. Self- talk: A self-supervised commutative training diagram to comprehend 3d talking faces. In Proceedings of the 31st ACM International Conference on Multimedia, page 5292–5301, 2023. 5, 6, 7

    2023

  26. [34]

    Emotalk: Speech-driven emotional disentanglement for 3d face animation

    Ziqiao Peng, Haoyu Wu, Zhenbo Song, Hao Xu, Xi- angyu Zhu, Jun He, Hongyan Liu, and Zhaoxin Fan. Emotalk: Speech-driven emotional disentanglement for 3d face animation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 20687–20697, 2023. 1

    2023

  27. [35]

    Memory based fusion for multi-modal deep learning

    Darshana Priyasad, Tharindu Fernando, Simon Den- man, Sridha Sridharan, and Clinton Fookes. Memory based fusion for multi-modal deep learning. Informa- tion Fusion, 67:136–146, 2021. 4

    2021

  28. [36]

    Meshtalk: 3d face animation from speech using cross-modality dis- entanglement

    Alexander Richard, Michael Zollh¨ofer, Yandong Wen, Fernando de la Torre, and Yaser Sheikh. Meshtalk: 3d face animation from speech using cross-modality dis- entanglement. In IEEE/CVF International Conference on Computer Vision (ICCV), pages 1153–1162, 2021. 1, 3, 7

    2021

  29. [37]

    Toward accurate dy- namic time warping in linear time and space.Intelligent Data Analysis, 11(5):561–580, 2007

    Stan Salvador and Philip Chan. Toward accurate dy- namic time warping in linear time and space.Intelligent Data Analysis, 11(5):561–580, 2007. 7

    2007

  30. [38]

    Towards end-to-end prosody trans- fer for expressive speech synthesis with tacotron

    RJ Skerry-Ryan, Eric Battenberg, Ying Xiao, Yuxuan Wang, Daisy Stanton, Joel Shor, Ron Weiss, Rob Clark, and Rif A Saurous. Towards end-to-end prosody trans- fer for expressive speech synthesis with tacotron. In international conference on machine learning, pages 4693–4702. PM...

    2018

  31. [39]

    Diffposetalk: Speech-driven stylistic 3d facial anima- tion and head pose generation via diffusion models

    Zhiyao Sun, Tian Lv, Sheng Ye, Matthieu Lin, Jenny Sheng, Yu-Hui Wen, Minjing Yu, and Yong-jin Liu. Diffposetalk: Speech-driven stylistic 3d facial anima- tion and head pose generation via diffusion models. ACM Transactions on Graphics (TOG) , 43(4):1–9,

  32. [40]

    Emmn: Emotional mo- tion memory network for audio-driven emotional talk- ing face generation

    Shuai Tan, Bin Ji, and Ye Pan. Emmn: Emotional mo- tion memory network for audio-driven emotional talk- ing face generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22146–22156, 2023. 4

    2023

  33. [41]

    Imitator: Personalized speech-driven 3d facial animation

    Balamurugan Thambiraja, Ikhsanul Habibie, Sadegh Aliakbarian, Darren Cosker, Christian Theobalt, and Justus Thies. Imitator: Personalized speech-driven 3d facial animation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 20621–20631, 20...

    2023

  34. [42]

    Visu- alizing data using t-sne

    Laurens Van der Maaten and Geoffrey Hinton. Visu- alizing data using t-sne. Journal of machine learning research, 9(11), 2008. 8, 5

    2008

  35. [43]

    Memory networks

    Jason Weston, Sumit Chopra, and Antoine Bordes. Memory networks. arXiv preprint arXiv:1410.3916,

    arXiv

  36. [44]

    Speech-driven 3d face animation with composite and regional facial movements

    Haozhe Wu, Songtao Zhou, Jia Jia, Junliang Xing, Qi Wen, and Xiang Wen. Speech-driven 3d face animation with composite and regional facial movements. In Proceedings of the 31st ACM International Conference on Multimedia, page 6822–6830, New York, NY , USA,

  37. [45]

    Codetalker: Speech-driven 3d facial animation with discrete mo- tion prior

    Jinbo Xing, Menghan Xia, Yuechen Zhang, Xiaodong Cun, Jue Wang, and Tien-Tsin Wong. Codetalker: Speech-driven 3d facial animation with discrete mo- tion prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12780–12790, 2023. 1, 3,...

    2023

  38. [46]

    Kmtalk: Speech-driven 3d facial animation with key motion embedding

    Zhihao Xu, Shengjie Gong, Jiapeng Tang, Lingyu Liang, Yining Huang, Haojie Li, and Shuangping Huang. Kmtalk: Speech-driven 3d facial animation with key motion embedding. In European Conference on Computer Vision, pages 236–253. Springer, 2024. 1

    2024

  39. [47]

    Probabilistic speech-driven 3d facial motion synthesis: new bench- marks methods and applications

    Karren D Yang, Anurag Ranjan, Jen-Hao Rick Chang, Raviteja Vemulapalli, and Oncel Tuzel. Probabilistic speech-driven 3d facial motion synthesis: new bench- marks methods and applications. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pag...

    2024

  40. [48]

    Tianyu Yang and Antoni B. Chan. Learning Dynamic Memory Networks for Object Tracking. In ECCV,

  41. [49]

    Audio-driven talking face video genera- tion with learning-based personalized head pose

    Ran Yi, Zipeng Ye, Juyong Zhang, Hujun Bao, and Yong-Jin Liu. Audio-driven talking face video genera- tion with learning-based personalized head pose. arXiv preprint arXiv:2002.10137, 2020. 4

    2002 arXiv

  42. [50]

    Fake news detection via multi- modal topic memory network

    Long Ying, Hui Yu, Jinguang Wang, Yongze Ji, and Shengsheng Qian. Fake news detection via multi- modal topic memory network. IEEE Access, 9:132818– 132829, 2021. 4

    2021

  43. [51]

    Media2face: Co-speech facial animation generation with multi-modality guidance

    Qingcheng Zhao, Pengyu Long, Qixuan Zhang, Dafei Qin, Han Liang, Longwen Zhang, Yingliang Zhang, Jingyi Yu, and Lan Xu. Media2face: Co-speech facial animation generation with multi-modality guidance. In ACM SIGGRAPH 2024 Conference Papers, New York, NY , USA, 2024. Association...

    2024

  44. [52]

    Inflated episodic memory with region self-attention for long-tailed visual recog- nition

    Linchao Zhu and Yi Yang. Inflated episodic memory with region self-attention for long-tailed visual recog- nition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4344– 4353, 2020. 4

    2020

  45. [53]

    Kangaroo

    Yixiang Zhuang, Baoping Cheng, Yao Cheng, Yuntao Jin, Renshuai Liu, Chengyang Li, Xuan Cheng, Jing Liao, and Juncong Lin. Learn2talk: 3d talking face learns from 2d talking face. IEEE Transactions on Visualization and Computer Graphics, 2024. 1 MemoryTalker: Personalized Speec...

    2024

  46. [55]

    /a:/” and “/w/

    to show that our method reflects speaking styles in the audio, contrasting it with existing approaches (see Figure S8). Figure S8 (a) visualizes the motion feature synthesized during inference when using one-hot encoding [12]. As in [12], since there are not able to know one-h...

  47. [2023]

    Association for Computing Machinery. 2, 3, 1

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.