Pith. sign in

REVIEW 4 major objections 5 minor 57 references

DualTalk: Dual-Speaker Interaction for 3D Talking Head Conversations

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read DualTalk claims that one unified model can generate both speaking and listening behaviors for 3D avatars across multi-round conversations, backed by a 50-hour dual-speaker dataset.

desk verdict A genuinely new task and dataset for dual-speaker 3D talking heads, but the quantitative claims rest on an unvalidated automatic annotation pipeline. read the letter →

arxiv 2505.18096 v2 pith:JHOEKNRC submitted 2025-05-23 cs.CV cs.SDeess.AS

Ziqiao Peng , Yanbo Fan , Haoyu Wu , Xuan Wang , Hongyan Liu , Jun He , Zhaoxin Fan This is my paper
classification cs.CVcs.SDeess.AS
keywords 3Dtalkingheadgenerationdual-speakerinteractionlistenermodelingnon-verbalfeedbackmulti-roundconversationaudio-visualdatasetblendshapeanimationfacial
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes a new task—multi-round dual-speaker interaction for 3D talking head generation—and argues that a single model should handle both roles a person plays in conversation: speaking with lip-synchronized motion and listening with contextually appropriate nods, smiles, and other non-verbal feedback. To test that claim, it introduces DualTalk, a unified network that takes one speaker's audio and facial motion plus the other speaker's audio and produces the other speaker's head motion, so role switches happen inside one model rather than by stitching together a speaker-only and a listener-only system. The paper also contributes a 50-hour dataset of more than 1,000 people in two-person conversations, built with automatic speech separation, face tracking, mesh fitting, and speaker diarization, which it presents as the first 3D facial mesh dataset aimed at face-to-face multi-round interactions. Experiments report that DualTalk beats separate speaker-only and listener-only baselines on realism, synchronization, and diversity metrics and runs in real time. If the claim holds, conversational avatars could sustain extended back-and-forth interactions instead of alternating between canned speaking and detached listening.

What carries the argument

The load-bearing mechanism is the conditional mapping $\hat{M}_B = f(A_A, M_A, A_B)$ implemented by four stacked modules: a dual-speaker joint encoder (separate audio encoders and a blendshape encoder projecting all inputs into one 256-dimensional feature space), a cross-modal temporal enhancer (cross-attention plus a bidirectional LSTM that aligns the two modalities over time), a dual-speaker interaction module (a transformer encoder-decoder with a modal-alignment attention mask that lets each speaker's features condition the other's), and an expressive synthesis module (an adaptive modulation layer that adjusts the final blendshape output). A second load-bearing mechanism is the dataset-construction pipeline, which turns raw two-person interview videos into training targets by automatically separating each speaker's audio, detecting and tracking faces, fitting 3D meshes, and diarizing speaker turns. The model is trained with a blendshape reconstruction loss plus a velocity loss that penalizes jerky frame-to-frame motion.

What would settle it

Compute an independent lip-sync score between each speaker's separated audio and the visible lip motion in the original videos; if the scores on the training set are no better than those for randomly mismatched audio-video pairs, the automatic separation that grounds both training and evaluation is too noisy to support the paper's quantitative claims.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that jointly modeling speaker and listener behavior in one network produces more natural and expressive 3D conversations than modeling either role alone. The task is formalized as learning $\hat{M}_B = f(A_A, M_A, A_B)$, where $A_A$ and $A_B$ are the two participants' audio, $M_A$ is the visible speaker's facial motion in blendshape coefficients, and $\hat{M}_B$ is the generated listener's motion. DualTalk implements this with four modules: a dual-speaker joint encoder that projects both audio streams and the visible speaker's blendshapes into a shared space; a cross-modal temporal enhancer that aligns audio and visual features over time; a dual-speaker interaction module that exchanges information between the two speakers via a transformer with a modal-alignment attention mask; and an expressive synthesis module that modulates the final expression output. The accompanying dataset supplies the paired audio-visual ground truth: roughly 5,858 clips totaling 50 hours from 1,052 identities, with separate audio streams per speaker and 56-dimension blendshape targets, split into training, test, and out-of-distribution sets. Across the reported metrics and a user study, DualTalk is claimed to significantly improve naturalness, synchronization, and diversity compared with speaker-only, listener-only, and concatenated baselines.

Load-bearing premise

The whole training and evaluation chain rests on the assumption that the automatically produced ground truth—separated audio, tracked faces, fitted 3D meshes, and diarized turns—is accurate enough to serve as the target for lip sync and listener behavior; the paper reports no human validation of that pipeline.

Editorial extensions

If this is right

  • A single DualTalk model can replace the two-stage pipeline of a speaker-only generator and a listener-only generator, removing the need to stitch disjoint outputs into one conversation.
  • Listener reactions are conditioned on the partner's audio and facial expressions, so the model can, in principle, produce different non-verbal feedback in response to different conversational cues across multiple turns.
  • On the new benchmark, DualTalk is reported to reduce expression error by more than half relative to the best baseline and to increase motion diversity by 40% compared with the strongest speaker-only baseline, while also improving lip sync and pose metrics.
  • The reported 0.03 seconds of compute per second of generated feedback means the approach could drive real-time avatar systems.
  • The dataset and benchmark give the field a shared testbed for dual-speaker, multi-round 3D conversation, including an out-of-distribution split with unseen speakers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same conditioning structure could extend to three or more participants by adding more audio and motion streams, although the paper states it currently handles only two-party conversations.
  • Editorial inference: because the dataset is built from public interview videos, the learned listening behavior will reflect the turn-taking and feedback norms of those videos; transfer to other cultures or settings is untested.
  • Editorial inference: a straightforward stress test of the dataset is to run an independent lip-sync or mesh-fidelity check on the automatically separated tracks; the quantitative claims assume those tracks are accurate ground truth.
  • Editorial inference: the model's listener behavior is learned implicitly, so one could probe causal sensitivity by holding the listener's audio fixed, varying the partner's expression, and measuring whether the generated listening motion changes in a meaningful way.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a new task, multi-round dual-speaker interaction for 3D talking head generation, and a model called DualTalk that takes Speaker-A's audio, Speaker-A's facial motion, and Speaker-B's audio as inputs and generates Speaker-B's facial motion, switching between speaking and listening roles. The authors also construct a 50-hour dataset of dyadic conversations from YouTube interviews with automatically extracted FLAME blendshape coefficients and separated audio streams, and they evaluate DualTalk against speaker-only and listener-only baselines using FD, P-FD, MSE, SID, rPCC, and a small user study. The central claim is that DualTalk significantly enhances the naturalness and expressiveness of 3D talking heads in dual-speaker conversations.

Significance. If the results are validated, the dataset is a potentially valuable community resource: it is the first 3D facial mesh dataset targeted at multi-round dyadic interaction, with over 1,000 identities and 50 hours of data, and the task formulation goes beyond single-role talking head generation. The model architecture, with a dual-speaker joint encoder, cross-modal temporal enhancer, and interaction module, is a reasonable design for conditioning one participant's motion on the other participant's speech and motion. However, the current paper does not establish the central claim because the evaluation protocol is asymmetric and the automatic annotation pipeline that produces both the training targets and the evaluation ground truth is not validated. The contribution is therefore best regarded as a promising direction and dataset proposal that needs substantially stronger experimental support.

major comments (4)
  1. [Section 3, Eq. (1)] The task definition uses Speaker-A's true motion M_A as an input and generates only Speaker-B's motion. This is not a joint dual-speaker generation framework; it is a one-sided conditional generation model with oracle access to the partner's ground-truth motion. In a genuine two-agent conversation, both participants' motions would need to be generated, and this model cannot be applied unless one participant's motion is already known. The paper should either clearly reframe the contribution as 'reactive listener generation given the speaker's true motion' or demonstrate a closed-loop setting where both motions are generated and the resulting interaction remains coherent.
  2. [Section 4.6 and Supplementary Section 2] The dataset construction pipeline is fully automatic and completely unvalidated: TransNet V2 shot detection, IIANet speech separation, Pyannote diarization, MediaPipe face tracking, and Spectre mesh fitting are applied at scale, with only an unspecified 'abnormal coefficients' filter. No quantitative validation of separation fidelity, diarization accuracy, or mesh-fitting error is reported. Because all metrics in Table 2 (FD, P-FD, MSE, SID, rPCC) are computed against these same automatically generated blendshape sequences, the low error values could reflect reproduction of annotation artifacts such as crosstalk, misassigned speaker turns, or jittery expression coefficients rather than genuinely natural conversational dynamics. The authors should report per-stage accuracy measures (e.g., speech separation SNR, diarization DER, mesh vertex error against manual fits) and ideally a human validation of a sample of the labels.
  3. [Section 5.1, Table 2] The comparison against baselines is asymmetric and therefore the reported margins are not informative. DualTalk is given both participants' audio and Speaker-A's true motion, while the single-role baselines (FaceFormer, CodeTalker, EmoTalk, SelfTalk, L2L) are not designed to consume all of these inputs and are retrained on a task for which they only see partial information. The paper also states that concatenated outputs of speaker-only and listener-only models 'lead to inferior results' but provides no table, figure, or numeric value for this comparison. A controlled experiment is needed: for example, all methods should receive the same available inputs where the architecture permits, and the concatenated baseline should be fully reported. An additional ablation of DualTalk that removes Speaker-A's motion and audio would isolate how much of the gain comes from the extra inputs rather than from the proposed architecture.
  4. [Section 5.2, Table 5] The user study is too small and lacks any statistical support for the claim that DualTalk 'outperforms previous methods across all evaluations.' With 30 clips and 30 raters, there are no confidence intervals, significance tests, or inter-rater agreement measures reported. The paper should include per-criteria means with standard errors, a paired significance test (e.g., Wilcoxon signed-rank), and ideally multiple independent rating rounds. Without this, the qualitative claim of perceptual superiority is not established.
minor comments (5)
  1. [Conclusion] The word 'muti-round' in the first sentence of Section 6 should be corrected to 'multi-round.'
  2. [Section 4.5, Eq. (10)] The modulation factor alpha is not specified as a learned scalar, a fixed hyperparameter, or a per-sample predicted value. Please clarify how alpha is set and whether it is included in the ablation 'w/o Adaptive Expression Modulation.'
  3. [Section 5.1] The claim of 'over 50% improvement in expression accuracy' is not tied to a specific metric; from Table 2 it appears to refer to MSE or FD, but the text should state the exact metric and the comparison value.
  4. [Supplementary Section 3] The FD and P-FD metrics require a pretrained motion encoder, but the encoder architecture and training data are not described. The SID metric depends on k-means with k=40, yet no variance or stability analysis is reported. Please specify these details so the metrics are reproducible.
  5. [Section 4.6] The text calls the dataset 'dual-channel audio' while the pipeline uses visual-guided speech separation to isolate each speaker's voice. Please clarify whether the original recordings have separate microphone channels or whether the two audio streams are entirely a product of IIANet.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: DualTalk's architecture and evaluations are self-contained, with external VOCASET and ViCo benchmarks providing independent support.

full rationale

DualTalk is a neural architecture and dataset paper; its claimed contribution is empirical rather than a derived mathematical result. The training objective (Supplementary Eqs. 13-17) is a standard MSE blendshape reconstruction plus velocity loss against ground-truth coefficients, and the reported metrics (FD, P-FD, MSE, SID, rPCC) are not optimized for. P-FD and rPCC are newly introduced evaluation statistics, but they are not training losses, so they do not create a fitted-input-called-prediction loop. The self-citations (e.g., SelfTalk [33], EmoTalk [34]) appear as related work and as baselines retrained on the same data; they are not load-bearing premises, and no uniqueness theorem or architecture-fixing ansatz is imported from the authors' prior work. The strongest caveat is in Section 4.6 and Supplementary Section 2: the DualTalk dataset ground truth is produced by an automatic pipeline (TransNet V2 shot detection, IIANet speech separation, Pyannote diarization, MediaPipe tracking, Spectre mesh fitting) with no reported validation of separated-audio fidelity or mesh-tracking accuracy. This is a real external-validity and annotation-noise risk for the in-house benchmark, but it is not circular: all baselines are evaluated against the same labels, and the model is additionally tested on external VOCASET and ViCo datasets, which provide independent evidence. The Supplementary Limitations section (Sec. 5) notes the model is dyadic-only and lacks emotional controllability; neither statement introduces circularity. No specific equation-level reduction or fitted-parameter-as-prediction step can be quoted, so the appropriate finding is no significant circularity.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claim depends on the standard assumption that a feed-forward network (with transformers, LSTM, cross-attention) can map multimodal cues to blendshapes, on the domain assumption that Wav2Vec and FLAME features retain the speech and expression information, and on the quality of the automatically constructed dataset. There is one explicit scalar (alpha in Eq. 10) whose value is not reported. No new physical entities are introduced.

free parameters (1)
  • alpha (adaptive expression modulation factor)
    Equation (10) applies alpha times Mod(D) to the decoder output; the paper does not state whether alpha is learned or hand-set, but it directly controls the strength of the modulation that the ablation study credits with improving expressiveness.
assumptions (4)
  • domain assumption Wav2Vec 2.0 audio embeddings contain sufficient speech information for lip synchronization
    The model relies on pretrained audio features to drive the mouth and expression synthesis.
  • domain assumption FLAME blendshape coefficients (b=56) faithfully represent the facial motion needed to reconstruct natural expressions and head pose
    The entire pipeline outputs blendshapes; any under-determination in the mesh fitting to monocular YouTube video becomes an error in the training target.
  • domain assumption The YouTube-derived corpus, with average 2.5 rounds per clip, provides a balanced distribution of speaker and listener turns
    If the diarization or speech-separation mislabels turns, the model may learn incorrect role transitions, and the round-count statistics in Figure 4 would be distorted.
  • domain assumption The new metrics P-FD and rPCC reflect human-perceived coherence of dyadic interaction
    The paper does not validate P-FD or rPCC against human judgments; they are introduced by the authors and used as evidence.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DualTalk: Dual-Speaker Interaction for 3D Talking Head Conversations." pith.science (2026). https://pith.science/paper/JHOEKNRC

@misc{pith2026250518096,
  author       = {Pith},
  title        = {Pith review of: DualTalk: Dual-Speaker Interaction for 3D Talking Head Conversations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JHOEKNRC}},
  note         = {Machine review of arXiv:2505.18096}
}
read the original abstract

In face-to-face conversations, individuals need to switch between speaking and listening roles seamlessly. Existing 3D talking head generation models focus solely on speaking or listening, neglecting the natural dynamics of interactive conversation, which leads to unnatural interactions and awkward transitions. To address this issue, we propose a new task -- multi-round dual-speaker interaction for 3D talking head generation -- which requires models to handle and generate both speaking and listening behaviors in continuous conversation. To solve this task, we introduce DualTalk, a novel unified framework that integrates the dynamic behaviors of speakers and listeners to simulate realistic and coherent dialogue interactions. This framework not only synthesizes lifelike talking heads when speaking but also generates continuous and vivid non-verbal feedback when listening, effectively capturing the interplay between the roles. We also create a new dataset featuring 50 hours of multi-round conversations with over 1,000 characters, where participants continuously switch between speaking and listening roles. Extensive experiments demonstrate that our method significantly enhances the naturalness and expressiveness of 3D talking heads in dual-speaker conversations. We recommend watching the supplementary video: https://ziqiaopeng.github.io/dualtalk.

Figures

Figures reproduced from arXiv: 2505.18096 by the authors.

Figure 1
Figure 1. Comparison of single-role models (Speaker-Only and Listener-Only) with DualTalk. Unlike single-role models, which lack key [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of DualTalk. DualTalk consists of four components: (a) Dual-Speaker Joint Encoder, (b) Cross-Modal Temporal Enhancer, (c) Dual-Speaker Interaction Module, and (d) Expressive Synthesis Module, enabling the generation of smooth and natural dual-speaker interactions. 2.2. Listener Modeling and Non-Verbal Feedback A complementary research area is the modeling of non￾verbal listener behaviors [15, 22, 23, 25, 28… view at source ↗
Figure 3
Figure 3. Dataset construction pipeline. The pipeline takes raw two-speaker videos and paired audio as input. It outputs segmented video clips, isolated audio streams for each speaker, 3D facial mesh data, and speaker round count statistics, providing high-quality, synchronized multimodal data for training. To effectively align these multimodal features, we in￾troduce a Modal Alignment Attention mechanism using an alignment m… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Distribution of conversation rounds in DualTalk dataset and example samples. Datasets Duration Identities Interaction Multi-Round Conversations VOCASET [7] 0.5h 12 ✗ ✗ BIWI [11] 1.44h 14 ✗ ✗ ViCO [49] 1.6h 92 ✓ ✗ L2L [27] 72h 6 ✓ ✗ Lm listener [28] 7h 4 ✓ ✗ RealTalk [1…
Figure 5
Figure 5. Figure 5: Qualitative comparison of speaking and listening states. The left side shows facial expressions in the speaking state, with DualTalk achieving more accurate lip movements compared to other methods. The right side shows expressions in the listening state, where DualTalk…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

57 extracted references · 43 canonical work pages

  1. [1]

    wav2vec 2.0: A framework for self-supervised learning of speech representations.Advances in neural infor- mation processing systems, 33:12449–12460, 2020

    Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations.Advances in neural infor- mation processing systems, 33:12449–12460, 2020. 4, 12

    work page 2020

  2. [2]

    High-fidelity fa- cial avatar reconstruction from monocular video with gen- erative priors

    Yunpeng Bai, Yanbo Fan, Xuan Wang, Yong Zhang, Jingx- iang Sun, Chun Yuan, and Ying Shan. High-fidelity fa- cial avatar reconstruction from monocular video with gen- erative priors. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4541– 4551, 2023. 1

    work page 2023

  3. [3]

    Pyannote

    Herv ´e Bredin, Ruiqing Yin, Juan Manuel Coria, Gregory Gelly, Pavel Korshunov, Marvin Lavechin, Diego Fustes, Hadrien Titeux, Wassim Bouaziz, and Marie-Philippe Gill. Pyannote. audio: neural building blocks for speaker diariza- tion. InICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7124–7128. IEE...

    work page 2020

  4. [4]

    Expressive speech-driven facial animation.ACM Transactions on Graphics (TOG), 24(4):1283–1302, 2005

    Yong Cao, Wen C Tien, Petros Faloutsos, and Fr ´ed´eric Pighin. Expressive speech-driven facial animation.ACM Transactions on Graphics (TOG), 24(4):1283–1302, 2005. 2

    work page 2005

  5. [5]

    Human conversation as a system framework: Designing embodied conversational agents.Em- bodied conversational agents, pages 29–63, 2000

    Justine Cassell, Tim Bickmore, Lee Campbell, Hannes Vil- hjalmsson, Hao Yan, et al. Human conversation as a system framework: Designing embodied conversational agents.Em- bodied conversational agents, pages 29–63, 2000. 1

    work page 2000

  6. [6]

    Cafe-talk: Generating 3d talking face animation with mul- timodal coarse-and fine-grained control

    Hejia Chen, Haoxian Zhang, Shoulong Zhang, Xiaoqiang Liu, Sisi Zhuang, Pengfei Wan, Di ZHANG, Shuai Li, et al. Cafe-talk: Generating 3d talking face animation with mul- timodal coarse-and fine-grained control. InThe Thirteenth International Conference on Learning Representations. 2

    work page

  7. [7]

    Capture, learning, and synthe- sis of 3d speaking styles

    Daniel Cudeiro, Timo Bolkart, Cassidy Laidlaw, Anurag Ranjan, and Michael J Black. Capture, learning, and synthe- sis of 3d speaking styles. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10101–10111, 2019. 1, 5, 6

    work page 2019

  8. [8]

    Stephan Diederich, Alfred Benedikt Brendel, Stefan Morana, and Lutz Kolbe. On the design of and interaction with con- versational agents: An organizing and assessing review of human-computer interaction research.Journal of the Asso- ciation for Information Systems, 23(1):96–138, 2022. 1

    work page 2022

Show all 57 references
  1. [9]

    Unitalker: Scaling up audio-driven 3d facial animation through a unified model.arXiv preprint arXiv:2408.00762,

    Xiangyu Fan, Jiaqi Li, Zhiqian Lin, Weiye Xiao, and Lei Yang. Unitalker: Scaling up audio-driven 3d facial animation through a unified model.arXiv preprint arXiv:2408.00762,

    arXiv

  2. [10]

    Faceformer: Speech-driven 3d facial anima- tion with transformers

    Yingruo Fan, Zhaojiang Lin, Jun Saito, Wenping Wang, and Taku Komura. Faceformer: Speech-driven 3d facial anima- tion with transformers. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18770–18780, 2022. 1, 2, 5, 6, 7, 8, 12

    2022

  3. [11]

    A 3-d audio-visual corpus of af- fective communication.IEEE Transactions on Multimedia, 12(6):591–598, 2010

    Gabriele Fanelli, Juergen Gall, Harald Romsdorfer, Thibaut Weise, and Luc Van Gool. A 3-d audio-visual corpus of af- fective communication.IEEE Transactions on Multimedia, 12(6):591–598, 2010. 5

    2010

  4. [12]

    Affective faces for goal-driven dyadic communication.arXiv preprint arXiv:2301.10939, 2023

    Scott Geng, Revant Teotia, Purva Tendulkar, Sachit Menon, and Carl V ondrick. Affective faces for goal-driven dyadic communication.arXiv preprint arXiv:2301.10939, 2023. 5, 13

    2023 arXiv

  5. [13]

    From pixels to portraits: A comprehensive survey of talking head generation tech- niques and applications.arXiv preprint arXiv:2308.16041,

    Shreyank N Gowda, Dheeraj Pandey, and Shashank Narayana Gowda. From pixels to portraits: A comprehensive survey of talking head generation tech- niques and applications.arXiv preprint arXiv:2308.16041,

    arXiv

  6. [14]

    Long short-term memory.Neural Computation MIT-Press, 1997

    S Hochreiter. Long short-term memory.Neural Computation MIT-Press, 1997. 4, 12

    1997

  7. [15]

    Toward rnn based micro non-verbal behavior generation for virtual listener agents

    Hung-Hsuan Huang, Masato Fukuda, and Toyoaki Nishida. Toward rnn based micro non-verbal behavior generation for virtual listener agents. InSocial Computing and Social Me- dia. Design, Human Behavior and Analytics: 11th Inter- national Conference, SCSM 2019, Held as Part of the...

    2019

  8. [16]

    Audio-driven facial animation by joint end- to-end learning of pose and emotion.ACM Transactions on Graphics (TOG), 36(4):1–12, 2017

    Tero Karras, Timo Aila, Samuli Laine, Antti Herva, and Jaakko Lehtinen. Audio-driven facial animation by joint end- to-end learning of pose and emotion.ACM Transactions on Graphics (TOG), 36(4):1–12, 2017. 2

    2017

  9. [17]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization.CoRR, abs/1412.6980, 2014. 13

    2014 arXiv

  10. [18]

    Iianet: An intra-and inter-modality attention network for audio- visual speech separation

    Kai Li, Runxuan Yang, Fuchun Sun, and Xiaolin Hu. Iianet: An intra-and inter-modality attention network for audio- visual speech separation. InForty-first International Con- ference on Machine Learning, 2024. 13

    2024

  11. [19]

    Learning a model of facial shape and expression from 4d scans.ACM Trans

    Tianye Li, Timo Bolkart, Michael J Black, Hao Li, and Javier Romero. Learning a model of facial shape and expression from 4d scans.ACM Trans. Graph., 36(6):194–1, 2017. 6

    2017

  12. [20]

    One-shot high-fidelity talking- head synthesis with deformable neural radiance field

    Weichuang Li, Longhao Zhang, Dong Wang, Bin Zhao, Zhi- gang Wang, Mulin Chen, Bang Zhang, Zhongjian Wang, Liefeng Bo, and Xuelong Li. One-shot high-fidelity talking- head synthesis with deformable neural radiance field. InPro- ceedings of the IEEE/CVF Conference on Computer Vi...

    2023

  13. [21]

    Proactive con- versational agents in the post-chatgpt world

    Lizi Liao, Grace Hui Yang, and Chirag Shah. Proactive con- versational agents in the post-chatgpt world. InProceed- ings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 3452–3455, 2023. 1

    2023

  14. [22]

    Mfr-net: Multi-faceted responsive listening head generation via denoising diffusion model

    Jin Liu, Xi Wang, Xiaomeng Fu, Yesheng Chai, Cai Yu, Jiao Dai, and Jizhong Han. Mfr-net: Multi-faceted responsive listening head generation via denoising diffusion model. In Proceedings of the 31st ACM International Conference on Multimedia, pages 6734–6743, 2023. 3

    2023

  15. [23]

    Customlistener: Text-guided responsive inter- action for user-friendly listening head generation

    Xi Liu, Ying Guo, Cheng Zhen, Tong Li, Yingying Ao, and Pengfei Yan. Customlistener: Text-guided responsive inter- action for user-friendly listening head generation. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2415–2424, 2024. 2, 3

    2024

  16. [24]

    Medi- apipe: A framework for building perception pipelines.arXiv preprint arXiv:1906.08172, 2019

    Camillo Lugaresi, Jiuqiang Tang, Hadon Nash, Chris Mc- Clanahan, Esha Uboweja, Michael Hays, Fan Zhang, Chuo- Ling Chang, Ming Guang Yong, Juhyun Lee, et al. Medi- apipe: A framework for building perception pipelines.arXiv preprint arXiv:1906.08172, 2019. 13

    1906 arXiv

  17. [25]

    Reactface: Multiple appropri- ate facial reaction generation in dyadic interactions.arXiv preprint arXiv:2305.15748, 2023

    Cheng Luo, Siyang Song, Weicheng Xie, Micol Spitale, Lin- lin Shen, and Hatice Gunes. Reactface: Multiple appropri- ate facial reaction generation in dyadic interactions.arXiv preprint arXiv:2305.15748, 2023. 3

    2023 arXiv

  18. [26]

    Diffspeaker: Speech-driven 3d facial animation with diffusion transformer.arXiv preprint arXiv:2402.05712, 2024

    Zhiyuan Ma, Xiangyu Zhu, Guojun Qi, Chen Qian, Zhaox- iang Zhang, and Zhen Lei. Diffspeaker: Speech-driven 3d facial animation with diffusion transformer.arXiv preprint arXiv:2402.05712, 2024. 6

    2024 arXiv

  19. [27]

    Learning to listen: Modeling non-deterministic dyadic facial motion

    Evonne Ng, Hanbyul Joo, Liwen Hu, Hao Li, Trevor Darrell, Angjoo Kanazawa, and Shiry Ginosar. Learning to listen: Modeling non-deterministic dyadic facial motion. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20395–20405, 2022. 1,...

    2022

  20. [28]

    Can language models learn to listen? InProceedings of the IEEE/CVF In- ternational Conference on Computer Vision, pages 10083– 10093, 2023

    Evonne Ng, Sanjay Subramanian, Dan Klein, Angjoo Kanazawa, Trevor Darrell, and Shiry Ginosar. Can language models learn to listen? InProceedings of the IEEE/CVF In- ternational Conference on Computer Vision, pages 10083– 10093, 2023. 2, 3, 5

    2023

  21. [29]

    From audio to photoreal embodiment: Synthesizing humans in conversations

    Evonne Ng, Javier Romero, Timur Bagautdinov, Shaojie Bai, Trevor Darrell, Angjoo Kanazawa, and Alexander Richard. From audio to photoreal embodiment: Synthesizing humans in conversations. InProceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pa...

    2024

  22. [30]

    Real-time 3d talking head from a synthetic viseme dataset

    Arthur Niswar, Ee Ping Ong, Hong Thai Nguyen, and Zhiy- ong Huang. Real-time 3d talking head from a synthetic viseme dataset. InProceedings of the 8th International Con- ference on Virtual Reality Continuum and its Applications in Industry, pages 29–33, 2009. 1

    2009

  23. [31]

    Scantalk: 3d talking heads from unregistered scans.arXiv preprint arXiv:2403.10942, 2024

    Federico Nocentini, Thomas Besnier, Claudio Ferrari, Syl- vain Arguillere, Stefano Berretti, and Mohamed Daoudi. Scantalk: 3d talking heads from unregistered scans.arXiv preprint arXiv:2403.10942, 2024. 2

    2024 arXiv

  24. [32]

    Dpe: Dis- entanglement of pose and expression for general video por- trait editing

    Youxin Pang, Yong Zhang, Weize Quan, Yanbo Fan, Xi- aodong Cun, Ying Shan, and Dong-ming Yan. Dpe: Dis- entanglement of pose and expression for general video por- trait editing. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 427–436,

  25. [33]

    Selftalk: A self- supervised commutative training diagram to comprehend 3d talking faces

    Ziqiao Peng, Yihao Luo, Yue Shi, Hao Xu, Xiangyu Zhu, Hongyan Liu, Jun He, and Zhaoxin Fan. Selftalk: A self- supervised commutative training diagram to comprehend 3d talking faces. InProceedings of the 31st ACM International Conference on Multimedia, pages 5292–5301, 2023. 1, 2, 6

    2023

  26. [34]

    Emotalk: Speech-driven emotional disentanglement for 3d face anima- tion

    Ziqiao Peng, Haoyu Wu, Zhenbo Song, Hao Xu, Xiangyu Zhu, Jun He, Hongyan Liu, and Zhaoxin Fan. Emotalk: Speech-driven emotional disentanglement for 3d face anima- tion. InProceedings of the IEEE/CVF International Confer- ence on Computer Vision, pages 20687–20697, 2023. 1, 2, 6, 7, 8

    2023

  27. [35]

    Synctalk: The devil is in the synchronization for talking head synthesis

    Ziqiao Peng, Wentao Hu, Yue Shi, Xiangyu Zhu, Xiaomei Zhang, Hao Zhao, Jun He, Hongyan Liu, and Zhaoxin Fan. Synctalk: The devil is in the synchronization for talking head synthesis. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 666–676,

  28. [36]

    Meshtalk: 3d face an- imation from speech using cross-modality disentanglement

    Alexander Richard, Michael Zollh ¨ofer, Yandong Wen, Fer- nando De la Torre, and Yaser Sheikh. Meshtalk: 3d face an- imation from speech using cross-modality disentanglement. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 1173–1182, 2021. 1, 6

    2021

  29. [37]

    Emotional listener portrait: Neural lis- tener head generation with emotion

    Luchuan Song, Guojun Yin, Zhenchao Jin, Xiaoyi Dong, and Chenliang Xu. Emotional listener portrait: Neural lis- tener head generation with emotion. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 20839–20849, 2023. 1, 3

    2023

  30. [38]

    React2023: The first multiple appropriate facial reaction generation chal- lenge

    Siyang Song, Micol Spitale, Cheng Luo, Germ ´an Barquero, Cristina Palmero, Sergio Escalera, Michel Valstar, Tobias Baur, Fabien Ringeval, Elisabeth Andr ´e, et al. React2023: The first multiple appropriate facial reaction generation chal- lenge. InProceedings of the 31st ACM ...

    2023

  31. [39]

    Transnet v2: An effective deep network architecture for fast shot transition detection

    Tom ´aˇs Souˇcek and Jakub Loko ˇc. Transnet v2: An effective deep network architecture for fast shot transition detection. arXiv preprint arXiv:2008.04838, 2020. 13

    2008 arXiv

  32. [40]

    Laughtalk: Expressive 3d talking head generation with laughter

    Kim Sung-Bin, Lee Hyun, Da Hye Hong, Suekyeong Nam, Janghoon Ju, and Tae-Hyun Oh. Laughtalk: Expressive 3d talking head generation with laughter. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 6404–6413, 2024. 1

    2024

  33. [41]

    Imitator: Personalized speech-driven 3d facial animation

    Balamurugan Thambiraja, Ikhsanul Habibie, Sadegh Aliak- barian, Darren Cosker, Christian Theobalt, and Justus Thies. Imitator: Personalized speech-driven 3d facial animation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 20621–20631, 2023. 1

    2023

  34. [42]

    Dyadic interaction modeling for social behavior generation.arXiv preprint arXiv:2403.09069, 2024

    Minh Tran, Di Chang, Maksim Siniukov, and Mohammad Soleymani. Dyadic interaction modeling for social behavior generation.arXiv preprint arXiv:2403.09069, 2024. 2, 3, 7

    2024 arXiv

  35. [43]

    Attention is all you need.Advances in Neural Information Processing Systems, 2017

    A Vaswani. Attention is all you need.Advances in Neural Information Processing Systems, 2017. 12

    2017

  36. [44]

    Vgg-tex: A vivid geometry-guided facial texture estimation model for high fidelity monocular 3d face reconstruction.arXiv preprint arXiv:2409.09740, 2024

    Haoyu Wu, Ziqiao Peng, Xukun Zhou, Yunfei Cheng, Jun He, Hongyan Liu, and Zhaoxin Fan. Vgg-tex: A vivid geometry-guided facial texture estimation model for high fidelity monocular 3d face reconstruction.arXiv preprint arXiv:2409.09740, 2024. 1

    2024 arXiv

  37. [45]

    Autogen: Enabling next-gen llm ap- plications via multi-agent conversation framework.arXiv preprint arXiv:2308.08155, 2023

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. Autogen: Enabling next-gen llm ap- plications via multi-agent conversation framework.arXiv preprint arXiv:2308.08155, 2023. 1

    2023 arXiv

  38. [46]

    Codetalker: Speech-driven 3d facial animation with discrete motion prior

    Jinbo Xing, Menghan Xia, Yuechen Zhang, Xiaodong Cun, Jue Wang, and Tien-Tsin Wong. Codetalker: Speech-driven 3d facial animation with discrete motion prior. InProceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12780–12790, 2023. 1, 2, 6, 7, 8

    2023

  39. [47]

    Nofa: Nerf-based one-shot facial avatar recon- struction

    Wangbo Yu, Yanbo Fan, Yong Zhang, Xuan Wang, Fei Yin, Yunpeng Bai, Yan-Pei Cao, Ying Shan, Yang Wu, Zhongqian Sun, et al. Nofa: Nerf-based one-shot facial avatar recon- struction. InACM SIGGRAPH 2023 conference proceedings, pages 1–12, 2023. 1

    2023

  40. [48]

    Human-computer interaction system: A survey of talking-head generation.Electronics, 12(1):218, 2023

    Rui Zhen, Wenchao Song, Qiang He, Juan Cao, Lei Shi, and Jia Luo. Human-computer interaction system: A survey of talking-head generation.Electronics, 12(1):218, 2023. 1

    2023

  41. [49]

    Responsive listening head generation: a benchmark dataset and baseline

    Mohan Zhou, Yalong Bai, Wei Zhang, Ting Yao, Tiejun Zhao, and Tao Mei. Responsive listening head generation: a benchmark dataset and baseline. InEuropean Conference on Computer Vision, pages 124–142. Springer, 2022. 3, 5, 7

    2022

  42. [50]

    Meta-learning empowered meta-face: Personalized speaking style adap- tation for audio-driven 3d talking face animation.arXiv preprint arXiv:2408.09357, 2024

    Xukun Zhou, Fengxin Li, Ziqiao Peng, Kejian Wu, Jun He, Biao Qin, Zhaoxin Fan, and Hongyan Liu. Meta-learning empowered meta-face: Personalized speaking style adap- tation for audio-driven 3d talking face animation.arXiv preprint arXiv:2408.09357, 2024. 1, 2

    2024 arXiv

  43. [51]

    Visemenet: Audio- driven animator-centric speech animation.ACM Transac- tions on Graphics (TOG), 37(4):1–10, 2018

    Yang Zhou, Zhan Xu, Chris Landreth, Evangelos Kaloger- akis, Subhransu Maji, and Karan Singh. Visemenet: Audio- driven animator-centric speech animation.ACM Transac- tions on Graphics (TOG), 37(4):1–10, 2018. 1 DualTalk: Dual-Speaker Interaction for 3D Talking Head Conversatio...

    2018

  44. [52]

    Network Architecture In this section, we provide comprehensive implementation details of our DualTalk framework

    Implementation Details 1.1. Network Architecture In this section, we provide comprehensive implementation details of our DualTalk framework. The framework con- sists of four main components: Dual-Speaker Joint En- coder, Cross-Modal Temporal Enhancer, Dual-Speaker In- teractio...

  45. [54]

    Here, we provide detailed in- formation about our data collection, processing procedures, and dataset statistics

    Dataset Details Our dataset collection and processing pipeline is designed to create a comprehensive and high-quality dataset for dual- speaker interaction modeling. Here, we provide detailed in- formation about our data collection, processing procedures, and dataset statistic...

    1920

  46. [55]

    Evaluation Metrics In this section, we provide detailed descriptions of the eval- uation metrics used to assess the performance of our Du- alTalk framework. These metrics are carefully selected to comprehensively evaluate different aspects of the generated conversational anima...

  47. [56]

    Ethics Considerations The development of DualTalk raises important ethical con- siderations, particularly regarding privacy, misuse, and po- tential societal impacts. The DualTalk dataset includes ex- tensive conversational data, and while publicly available sources were used,...

  48. [57]

    While DualTalk ex- cels in creating synchronized and natural two-speaker con- versations, it cannot yet handle multi-party interactions, which are common in real-world applications

    Limitations and Future Works The limitations of DualTalk primarily lie in its current fo- cus on dyadic interactions and the lack of precise emotional controllability in generated animations. While DualTalk ex- cels in creating synchronized and natural two-speaker con- versati...

  49. [512]

    The decoder follows a similar structure but includes additional cross- attention layers to integrate information from both speakers

    The Modal Alignment Attention layer, inspired by FaceFormer [10], uses a custom attention mask to ensure causal relationships in the temporal domain. The decoder follows a similar structure but includes additional cross- attention layers to integrate information from both spea...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.