{"id":"66f3cf83-6e52-40f8-9274-4727c997bf49","arxiv_id":"2507.20562","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A two-stage key-value memory model generates personalized 3D facial animation from audio alone, topping prior methods on FVE, LVE, FID, LDTW, and lip-max on VOCASET and BIWI.","lead":"MemoryTalker makes a 3D animated face speak in sync with audio while copying the speaker's individual mouth style, using only the voice recording as input. It first stores generic lip movements in a memory bank, then tweaks those movements using a style feature read from the audio, removing the need for identity labels or reference face scans at test time.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unseen-voice generalization is the central unsupported premise: the style encoder is trained on a small speaker pool and evaluated on a small test split of the same corpus, so the claim that MemoryTalker personalizes arbitrary new voices from audio alone is asserted, not demonstrated.","rationale":"The reader's weakest assumption is the correct load-bearing point: style/content disentanglement and transfer to unseen voices are not established. The reader's CONDITIONAL verdict is appropriate; my stress test does not move it, but it sharpens the condition. The held-out speaker and cross-dataset checks would settle whether the central claim is valid or whether the method should be described as speaker memorization rather than audio-only personalization. I do not see an internal inconsistency that would justify REJECT; the mechanism is coherent and the within-benchmark numbers are plausible. The absence of code and the missing DiffPoseTalk comparison are secondary to the generalization question.","tokens_in":18914,"tokens_out":6392,"duration_ms":77799,"concrete_test":"Retrain stage 2 with a leave-speaker-out protocol on VOCASET: hold out 4 of the 12 speakers from style-encoder training (or, stronger, from both stages), then evaluate FVE/LVE/LDTW separately for held-out and seen speakers. If held-out metrics degrade by more than the gap between MemoryTalker and UniTalker in Table 1, the style feature is memorizing training voices rather than generalizing. As a minimal complement, report per-speaker results on the existing VOCA-Test set and run a cross-dataset transfer (train on VOCASET, test on BIWI without fine-tuning) to check whether fs transfers to a different recording setup.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim—that a new voice can be animated with its characteristic mouth style using only audio—depends entirely on Section 3.2's style encoder Es and its triplet loss (Eq. 13). The paper's evidence does not establish the required transfer. Quantitatively, the reported FVE/LVE/LDTW numbers come from VOCA-Test/BIWI-Test-B, which are small fixed splits of the same datasets used to train Es; no per-held-out-speaker breakdown is given. The t-SNE plots (Fig. 6, Fig. S8) are the only direct evidence for style separation, and they show clusters of training/evaluation-pool speakers, not generalization to unseen voices. Because Lstyle is supervised with speaker identities, Es can satisfy it by encoding any speaker-correlated acoustic cue (pitch, timbre, rate) instead of articulatory style; since stage 2 freezes stage 1 and trains only Es on those speakers (Sec. 4.2), the gains in Table 1 could reflect memorized voice signatures. Moreover, fs is pooled over the whole mel-spectrogram, so without a content-invariance constraint it is not shown that fs is independent of phonetic content. The claim of audio-only personalization for arbitrary new speakers therefore rests on an untested assumption that the style feature transfers across voices.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"MemoryTalker proposes a two-stage framework for speech-driven 3D facial animation in which, at inference, only audio is required. In stage 1, a motion memory stores general facial motion features that are retrieved via text representations from a pretrained ASR encoder, aligned to motion features with an alignment loss. In stage 2, a speaker style encoder maps a mel-spectrogram to a style feature, a triplet loss separates speakers, and the memory slots are rescaled by style-dependent weights so that the recalled motion is personalized. The paper reports quantitative results on VOCASET and BIWI across FVE, LVE, FID, LDTW, and Lip-max, an ablation study, a memory-slot-size study, efficiency measurements, and a user study, claiming state-of-the-art performance without ID labels or mesh inputs at test time.","tokens_in":19198,"tokens_out":5211,"duration_ms":61931,"significance":"If the central claim holds, the practical contribution is clear: removing ID labels and reference meshes at inference makes personalized 3D facial animation more deployable. The two-stage memory design is well specified and the paper includes a reasonable amount of ablative and perceptual evidence. The method is also efficient in parameter count and inference time. However, the paper's headline claim of audio-only personalization for arbitrary unseen voices is not supported by the current evaluation design, which lacks a direct held-out-speaker generalization test and statistical confidence measures. The underlying idea is defensible and the gaps are addressable, so the appropriate path is a major revision.","major_comments":[{"comment":"The central claim that MemoryTalker personalizes an arbitrary new voice from audio alone is not established by the reported experiments. The style encoder Es is trained with a triplet loss whose classes are the training speakers, and the quantitative results in Table 1 are averaged over VOCA-Test/BIWI-Test-B without a per-held-out-speaker breakdown. The t-SNE in Fig. 6 and Fig. S8 clusters mostly the training-pool speakers, and Fig. S8 only compares encoding types, so the plots do not show that fs transfers to unseen voices. To support the central claim, please report leave-one-speaker-out experiments in which stage-2 training excludes the test speakers, per-speaker FVE/LVE on the held-out split, and a content-invariance experiment (e.g., same sentence across speakers versus different sentences for the same speaker, with style-feature retrieval or FVE as the metric). Without this evidence, the gains in Table 1 could reflect memorized voice signatures rather than a transferable articulatory style.","section":"Section 3.2, Eq. (12)-(13), and Table 1"},{"comment":"The quantitative comparison lacks statistical confidence measures for MemoryTalker itself. Table 1 reports point estimates with no standard deviations or confidence intervals, although Table 2 reports variability for the one-hot baselines. Several reported differences are small (e.g., BIWI FVE 0.901 vs. 0.919 and LDTW 0.107 vs. 0.109), so run-to-run or identity-to-identity variation could alter the conclusions. The user-study percentages in Table 6 are based on a modest number of participants and are not accompanied by any significance test. Please report means with standard deviations or confidence intervals across runs or test identities and provide a paired significance test such as bootstrap or Wilcoxon for the A/B preferences.","section":"Tables 1 and 6, Section 4.3 and 4.7"},{"comment":"The evaluation protocol for one-hot-based competitors is not aligned with the deployment setting claimed in the paper. The Fig. 4 caption states that the displayed one-hot encoding results use the ID with the lowest error, which gives those methods access to an oracle identity choice unavailable in practice, while MemoryTalker receives only audio. Please clarify whether the Table 1 numbers use this oracle selection and also report the conventional protocol of averaging over all training identities or using a fixed identity. Table S3 provides a useful seen-identity comparison for BIWI, but the VOCASET comparison should be made consistent across methods and clearly described so that the claimed advantage is not an artifact of the comparison protocol.","section":"Section 4.4 and Fig. 4 caption"}],"minor_comments":[{"comment":"When stating that quantitative evaluations were performed on VOCA-Test and BIWI-Test-B, please explicitly state whether the test identities were completely unseen during stage-2 training and give the speaker-level split; this is important for readers to interpret the personalization claim.","section":"Section 4.2"},{"comment":"The style weight ws contains a scalar term psi_1(fs) that rescales all memory slots; please ablate or visualize this scalar to show that it is not acting as a simple global amplitude gate rather than a per-slot style modulation.","section":"Eq. (15)"},{"comment":"The efficiency comparison reports a single inference-time value for each method; please state the batch size, sequence length, and hardware conditions under which the timings were measured, and report the standard deviation over repeated runs if the measurement is noisy.","section":"Table 3"},{"comment":"The main text says 33 subjects participated, while the supplement says 2 participants were removed after failing qualification questions; please report the final number of participants consistently in the main text and describe how many questions remained per participant after exclusions.","section":"Section 4.7 and Section E.2 of the supplement"},{"comment":"The memory-slot-size study reports only LVE; since the number of slots is a free parameter of the model, please report FVE, FID, and LDTW for the same sweep so that the choice of 32 slots is justified across all headline metrics.","section":"Figure S1"},{"comment":"The paper does not mention plans to release code or pretrained models; given the number of architectural details and hyperparameters, a public implementation would materially help reproducibility.","section":"Reproducibility"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern is the right one: the paper's central claim is audio-only transferable style, and the current evaluation does not directly demonstrate generalization to unseen voices. If the authors add a proper held-out-speaker evaluation and statistical rigor, the contribution could become solid. I would also ask the authors to check the novelty wording against the recent audio-driven stylization works cited in their own related work, since the claim of being the first to avoid all prior information at inference is stated more strongly than the evidence supports."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a competent, well-engineered paper that proposes a practical way to personalize 3D facial animation from audio alone, and the benchmark results are consistently strong. But the central claim—that it generalizes to arbitrary new voices—is never actually tested. The style encoder is trained on a small pool of speakers (12 on VOCASET, 14 on BIWI) and evaluated on the standard test splits of those same datasets, i.e. same identities, different sentences. The t-SNE plots show training speakers separating, not transfer to novel voices. So the headline is asserted, not demonstrated.\n\nWhat's new: the combination of a key-value motion memory with an audio-driven style encoder that rescales memory slots is, as far as I can tell, the first audio-only approach to personalization in this task. The two-stage training—first memorize speaker-neutral motion-text associations via an ASR model, then freeze everything and train only a mel-spectrogram style encoder with a triplet loss—is clean and plausible. The paper also does solid work: standard datasets, five metrics, ablations, a user study, and a comparison against both one-hot and mesh-based methods. The efficiency numbers (94M params, 7.8 ms per frame) are a nice practical plus.\n\nWeak spots, in proportion. The biggest one is the missing generalization experiment. A held-out speaker or cross-dataset evaluation would directly test the paper's raison d'être. Without it, the style feature might be picking up speaker-specific timbre or pitch rather than articulatory style, and a new voice could fail. The style feature is a single global pool over the mel-spectrogram, and there's no content-invariance constraint, so it may also encode phonetic content. These are real gaps, but they are fixable.\n\nAlso worth noting: no error bars for the proposed model's headline numbers, no statistical significance tests for the user study, and no comparison with DiffPoseTalk, a recent diffusion-based stylistic method. The comparison methods are somewhat disadvantaged by construction—one-hot methods can't represent unseen speakers, and mesh-based methods need extra inputs—but that's the point of the paper, so it's fair enough.\n\nOverall, the architecture is coherent and the internal ablations justify the two-stage design. I'd send this to peer review, but I'd ask the authors to either restrict the claim to seen speakers or add a genuine test of unseen-voice generalization. If they can show the latter, this is a strong paper; if not, it's still a useful incremental contribution.","headline":"Competent and clean two-stage design for audio-only 3D facial animation, but the headline claim of arbitrary new-voice generalization is untested.","tokens_in":19760,"tokens_out":3032,"would_cite":false,"duration_ms":32222,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MemoryTalker claims a speaker's mouth style — opening, pouting, articulation — can be read from audio and personalize 3D animation for unseen voices, with no labels or reference meshes at inference.","keywords":["speech-driven 3D facial animation","speaking style","audio-guided stylization","motion memory network","key-value memory","style-content disentanglement","personalized facial animation","VOCASET"],"falsifier":"Train on a proper subset of speakers (for example, 10 of VOCASET's 12) and animate the held-out two; if the generated lip motion for the unseen voices is no more accurate than a speaker-neutral baseline, or if their style features collapse onto training clusters in t-SNE rather than forming their own, the audio-only personalization claim is falsified. A cheaper probe on the same point: take one speaker saying the same sentence twice, once over-articulated and once mumbled, and check that the style feature — and the generated mouth-opening amplitude — tracks articulation effort rather than only speaker identity.","tokens_in":18688,"feed_emoji":"👄","tokens_out":13592,"duration_ms":128487,"temperature":0.7,"pith_summary":"Speech-driven 3D facial animation must do two things at once: move the lips in sync with the words, and move them the way that particular speaker moves them. Prior systems either need a speaker-identity label, which cannot describe a voice never seen in training, or a reference 3D facial mesh at inference, which is impractical for real applications. This paper claims both needs can be met from the audio signal alone. Its two-stage scheme first stores generic, speaker-neutral lip motion in a learnable motion memory keyed by a speech-recognition representation of the words, then rescales every memory slot by a 'speaking style' vector extracted from a mel-spectrogram of the same audio and trained with a triplet loss to pull same-speaker styles together. If the claim holds, an arbitrary new voice is enough to drive a personalized avatar in real time — about 120 frames per second with 94M parameters — and the paper reports the lowest error on every metric on both VOCASET and BIWI, including lip-region error $0.293$ vs $0.375$ for the previous best method.","feed_headline":"Audio alone drives speaker-specific 3D facial animation","feed_subtitle":"MemoryTalker reads mouth style from the voice itself, with no labels or meshes, beating prior models on both benchmarks.","key_machinery":"The load-bearing object is the key-value motion memory $\\mathbf{M}_m \\in \\mathbb{R}^{n\\times c}$, with 32 slots, which plays two roles across the two training stages. In stage 1 (Memorizing) it stores and retrieves general facial motion: a motion encoder maps vertex displacements into query features, while text representations from a pre-trained ASR encoder form a softmax key-address vector, and a KL-divergence loss aligns the two addressing distributions so that the same word retrieves canonical, speaker-neutral lip motion. In stage 2 (Animating) the style feature $f_s$ from a mel-spectrogram encoder converts each slot into a scaled version, $\\tilde{\\mathbf{M}}_m = \\{\\tilde{w}^i_s \\cdot s^i_m\\}_{i=1}^n$ with $\\tilde{\\mathbf{w}}_s = \\mathrm{sigmoid}(\\psi'_{\\to n}(f_s))\\cdot\\psi_{\\to 1}(f_s)$, so recall from the stylized memory — decoded by a Transformer together with the text feature — yields motion whose lip shapes carry that speaker's articulation habits. The mechanism matters because it separates content from style at the memory level itself: the ASR key decides what the mouth does, the style weights decide how strongly, and that separation is what allows generic motion and individual style to be learned in different stages and combined with audio as the only input.","core_discovery":"The central claim is that speaking style is a property of the audio waveform itself — not of an identity slot and not of a reference mesh — and that a network can learn to extract it and inject it into animation. MemoryTalker encodes the words with a pre-trained ASR encoder to address a 32-slot key-value motion memory and recall what the paper calls general motion: the canonical lip shapes that all speakers share for each phoneme. A separate style encoder maps a mel-spectrogram of the same audio to a style feature $f_s$, trained with a triplet loss $\\mathcal{L}_{\\mathrm{style}}$ so clips of the same speaker cluster together, and that feature rescales the memory slots (via $\\tilde{\\mathbf{w}}_s = \\mathrm{sigmoid}(\\psi'_{\\to n}(f_s))\\cdot\\psi_{\\to 1}(f_s)$) into a stylized memory $\\tilde{\\mathbf{M}}_m$ from which personalized motion is recalled and decoded alongside the text feature. The paper reports that this construction outperforms every compared method on every metric on both datasets — on VOCASET, face vertex error $0.506$ vs $0.570$ and lip vertex error $0.293$ vs $0.375$ for the previous best — wins its user study on lip-sync, realism, and speaking style, and runs at about 120 fps with 94M parameters, with audio as the only inference input.","pith_inferences":["Because content and style are separated at the memory level, the same recipe — a content key addressing shared slots, a style vector rescaling them — could transfer to adjacent generation tasks such as 2D talking faces, head pose, upper-face expression, or co-speech gesture; the paper itself does not claim this extension.","The paper does not run the decisive speaker-generalization ablation: hold entire speakers out of training (not just test sentences) and check whether each unseen voice's style feature still separates in the latent space and whether its animations beat a speaker-neutral baseline on lip error. The t-SNE evidence shows only that training speakers separate, which is necessary but not sufficient for th","The triplet style space is anchored to training-set identity, so an adversarial probe — the same sentence spoken over-articulated versus mumbled by the same voice — would reveal whether the encoder tracks articulation effort, which the animation claim needs, or merely voice timbre, which would also separate speakers in t-SNE.","If the style feature behaves as claimed, it could serve as a portable style prior beyond animation: given an arbitrary recording, the same $f_s$ could drive style transfer, dubbing, or voice-conversion pipelines that need lip motion matched to a target voice rather than to the source audio."],"forward_implications":["A new voice becomes a complete personalization signal: with audio as the only inference input, any unseen speaker can drive an avatar in their own articulation style, with no enrollment video, identity label, or mesh sequence.","The mechanism transfers across mesh topologies: trained on VOCASET's 5,023-vertex FLAME mesh, the same model evaluates directly on BIWI's denser 23,370-vertex mesh and still leads on every metric.","The system sits in an interactive range: 7.8 ms per frame and 94M parameters (about 120 fps on a single GPU), well below the cost of the transformer-based animators it is compared with.","The two-stage curriculum is itself part of the gain: two-stage memorization-then-stylization training beats end-to-end training (FVE $0.506$ vs $0.510$, LVE $0.293$ vs $0.303$), and ablations show that removing either the memory or the style stage degrades results.","Personalization re-uses a shared articulatory basis: key-address visualizations in the supplementary show the same memory slots activating for the same lip shapes across different speakers, so the stylized memory rescales common geometry knowledge rather than memorizing a per-identity template."],"supporting_citations":[{"why":"VOCASET: the paired audio/3D-scan dataset that defines the training and evaluation setup, the 12-speaker speaking-style problem, and the FLAME-mesh format.","marker":"[6]"},{"why":"BIWI: the second evaluation dataset, used to show the method transfers to a denser mesh topology and to emotional and neutral speech recordings.","marker":"[13]"},{"why":"FaceFormer: the transformer-based one-hot-identity baseline whose decoder structure the motion decoder follows and whose results the method must beat on the error metrics.","marker":"[12]"},{"why":"CodeTalker: the discrete-motion-prior, one-hot baseline used throughout the quantitative comparisons, ablations, and user study.","marker":"[45]"},{"why":"UniTalker: the previous state-of-the-art unified model that MemoryTalker outperforms on every metric on both VOCASET and BIWI.","marker":"[11]"},{"why":"Imitator: the reference-video personalization approach that defines the two-stage style-adaptation baseline and the practical alternative MemoryTalker's audio-only inference is designed to replace.","marker":"[41]"},{"why":"Mimic: the mesh-reference style-disentanglement method that represents the alternative requiring extra 3D facial motion at inference, contrasted in the supplementary.","marker":"[19]"},{"why":"HuBERT: the pre-trained ASR encoder that supplies the text representations used to address the motion memory for speaker-neutral content.","marker":"[18]"},{"why":"Du et al.: the convolutional architecture reused as the speaking style encoder $E_s$ that extracts the mel-spectrogram style feature.","marker":"[8]"}],"fun_headline_variants":["Voice-only stylized 3D faces beat meshes and labels","Audio's hidden style animates faces with no extra priors","MemoryTalker: style from speech alone, no meshes needed","Speaker style learned from audio, not identity or mesh","Audio-only 3D face animation that learns speaking style"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise, stated in Section 3.2, is that the mel-spectrogram style feature trained with a triplet loss isolates a speaker's articulation style from phonetic content and transfers to voices never heard in training; the figures show training speakers' features separating, but nothing in the paper demonstrates what happens to the style feature — or to the lip motion it produces — for a genuinely unseen voice.","fun_headline_variants_meta":{"raw":{"variants":["Voice-only stylized 3D faces beat meshes and labels","Audio's hidden style animates faces with no extra priors","MemoryTalker: style from speech alone, no meshes needed","Speaker style learned from audio, not identity or mesh","Audio-only 3D face animation that learns speaking style"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000205,"raw_usage":{"total_tokens":1447,"prompt_tokens":1050,"completion_tokens":397,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":666,"completion_tokens_details":{"reasoning_tokens":312}},"tokens_in":666,"tokens_out":397,"duration_ms":3960,"temperature":1.0,"reasoning_tokens":312,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:41:04.175141+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train on a proper subset of speakers (for example, 10 of VOCASET's 12) and animate the held-out two; if the generated lip motion for the unseen voices is no more accurate than a speaker-neutral baseline, or if their style features collapse onto training clusters in t-SNE rather than forming their own, the audio-only personalization claim is falsified. A cheaper probe on the same point: take one speaker saying the same sentence twice, once over-articulated and once mumbled, and check that the style feature — and the generated mouth-opening amplitude — tracks articulation effort rather than only speaker identity.","supporting_citations":[{"cited_title":"Capture, learning, and synthesis of 3D speaking styles","cited_arxiv_id":null,"evidence_quote":"VOCASET: the paired audio/3D-scan dataset that defines the training and evaluation setup, the 12-speaker speaking-style problem, and the FLAME-mesh format."},{"cited_title":"A 3-d audio-visual corpus of affective communication","cited_arxiv_id":null,"evidence_quote":"BIWI: the second evaluation dataset, used to show the method transfers to a denser mesh topology and to emotional and neutral speech recordings."},{"cited_title":"Faceformer: Speech-driven 3d facial animation with transformers","cited_arxiv_id":null,"evidence_quote":"FaceFormer: the transformer-based one-hot-identity baseline whose decoder structure the motion decoder follows and whose results the method must beat on the error metrics."},{"cited_title":"Codetalker: Speech-driven 3d facial animation with discrete mo- tion prior","cited_arxiv_id":null,"evidence_quote":"CodeTalker: the discrete-motion-prior, one-hot baseline used throughout the quantitative comparisons, ablations, and user study."},{"cited_title":"Unitalker: Scaling up audio-driven 3d facial animation through a unified model","cited_arxiv_id":null,"evidence_quote":"UniTalker: the previous state-of-the-art unified model that MemoryTalker outperforms on every metric on both VOCASET and BIWI."},{"cited_title":"Imitator: Personalized speech-driven 3d facial animation","cited_arxiv_id":null,"evidence_quote":"Imitator: the reference-video personalization approach that defines the two-stage style-adaptation baseline and the practical alternative MemoryTalker's audio-only inference is designed to replace."},{"cited_title":"Mimic: Speaking style disentanglement for speech- driven 3d facial animation","cited_arxiv_id":null,"evidence_quote":"Mimic: the mesh-reference style-disentanglement method that represents the alternative requiring extra 3D facial motion at inference, contrasted in the supplementary."},{"cited_title":"Hubert: Self-supervised speech representation learning by masked prediction of hidden units","cited_arxiv_id":null,"evidence_quote":"HuBERT: the pre-trained ASR encoder that supplies the text representations used to address the motion memory for speaker-neutral content."},{"cited_title":"Disentanglement of emotional style and speaker identity for expressive voice conversion","cited_arxiv_id":null,"evidence_quote":"Du et al.: the convolutional architecture reused as the speaking style encoder $E_s$ that extracts the mel-spectrogram style feature."}],"review_version":2}