Pith. sign in

REVIEW 4 major objections 6 minor 60 references

VividListener: Expressive and Controllable Listener Dynamics Modeling for Multi-Modal Responsive Interaction

T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read VividListener proposes that long-sequence listener head dynamics can be generated with fine-grained control from text descriptions and continuous emotional-intensity tags, and supports this with a new 1.4M-frame dyadic-conversation dataset.

desk verdict A potentially useful dataset and a plausible method, but the headline claim of fine-grained control is supported only by qualitative examples because the conditioning labels come from the target motions. read the letter →

arxiv 2504.21718 v2 pith:XDDPOBZH submitted 2025-04-30 cs.CV

Shiying Li , Xingqun Qi , Bingkun Yang , Chen Weile , Zezhao Tian , Muyi Sun , Qifeng Liu , Man Zhang
show 1 more author
Zhenan Sun
This is my paper · ORCID
classification cs.CV
keywords listenerdynamicsdyadicconversation3Dheadmotiongenerationdiffusiontransformeremotionintensitycontrolmulti-modalconditioningtext-guidedanimationvalence-arousal
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that listener head dynamics—the nodding, blinking, smiling, and other non-verbal reactions of the person who is not speaking—can be generated for long, multi-turn dialogues under fine-grained control. To do this, it assembles ListenerX, a dataset of 3D dyadic conversations with more than 1.4 million frames, each annotated with text descriptions of facial expression changes and continuous valence/arousal emotion-intensity tags. It then proposes VividListener, a diffusion-transformer generator that takes the speaker's motion and audio, the listener's text description, and intensity tags as conditions, and outputs an 8-second, 240-frame sequence of listener head motion. If correct, this would let virtual avatars display expressive, responsive reactions that shift with described emotion and with requested intensity, going beyond previous short-term, category-labeled listener models.

What carries the argument

The load-bearing components are the Responsive Interaction Module (RIM) and the Emotional Intensity Tags (EIT). RIM fuses speaker motion and audio via bidirectional multi-head attention, encodes listener text with a frozen vision-language text encoder, then computes a temporal similarity matrix between text embeddings and speaker features; adaptive pooling of this matrix produces blending weights that modulate the speaker features so the listener motion follows the described expression changes while keeping rhythm with the speaker. EIT injects continuous valence/arousal tags in two places: multiplied with the fused features as input, and used as the query in a cross-attention layer whose output is passed through a convolutional layer and an adaptive instance normalization layer to modulate intermediate listener motion features. Both feed a Diffusion Transformer denoiser trained with a denoising objective, an emotional-intensity prediction loss, and a velocity smoothness loss.

What would settle it

Take a fixed speaker clip and a fixed text description, then sweep the valence/arousal tags from their minimum to maximum while keeping everything else constant. If a held-out facial behavior recognizer does not show a monotonic increase in measured arousal or expression amplitude across the sweep, the claimed emotional-intensity control is not real. Independently, have two teams of human annotators re-describe a random sample of ListenerX clips and rate their intensity; low inter-annotator agreement on either would undermine the ground truth that both training and evaluation rely on.

Watch

Extended reading notes

Core claim

The central claim is that expressive, controllable listener dynamics for long dialogue sequences can be treated as a multi-modal conditioning problem and solved with a Diffusion Transformer, provided the missing ingredient—paired data with fine-grained annotations—is built at scale. On the collected ListenerX corpus, VividListener reports superior quantitative results across realism, synchrony, diversity, and accuracy metrics, including a 25.21% reduction in Fréchet distance over the best baseline under cross-scenario training. Ablations show that removing the text input, the intensity tags, the temporal-semantic interaction module, or the emotional-control layer degrades performance. The work also demonstrates conditional control with open-vocabulary descriptions and adjustable intensity values, and shows multi-turn role-switching results.

Load-bearing premise

The whole controllability argument rests on the trustworthiness of the ListenerX annotations: text descriptions produced by a vision-language model prompted with action units and then manually revised, and continuous emotion tags (pleasantness and excitement values) produced by an automatic facial-affect estimator. If those annotations are noisy or inconsistent, the learned mapping from text and tags to motion is compromised, and because the evaluation metrics use those same annotations as conditioning information, they do not independently verify the claimed control.

Editorial extensions

If this is right

  • Listener head motion can be generated in 8-second, 240-frame sequences while following text descriptions that describe emotion transitions, not just static labels.
  • Continuous valence/arousal tags provide a controllable dial for emotional intensity, so the same dialogue can be rendered calmer or more excited by changing tags.
  • Cross-scenario generalization improves dramatically: trained on casual daily conversations and tested on structured interviews, the method reports a 25.21% lower Fréchet distance than the sub-optimal baseline.
  • The ListenerX dataset, with 6,683 videos, 1.4M frames, text descriptions, and intensity tags, supports long-sequence multi-turn listener modeling for downstream avatar and interaction systems.
  • Ablations show that each condition matters: removing text or tags degrades semantic alignment and diversity, and removing the temporal-semantic interaction or emotional-control components lowers realism and control.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One consequence the paper leaves implicit: if released with its text and valence/arousal annotations, ListenerX could serve as a benchmark for other controllable generation tasks, including speaker-side generation, because it provides paired audio and 3D motion for both roles.
  • A testable extension would be to hold the speaker clip and text description fixed while sweeping the intensity tags across their range, then measure how monotonically the generated facial expression amplitude responds, isolating intensity control from speaker-imitation effects.
  • The metric design leaves room for a stronger controllability check: the reported rPCC and diversity metrics do not directly verify that the generated expression semantically matches the input text, so a separately trained text-expression alignment measure would make the claim easier to assess.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper addresses expressive and controllable listener head-dynamics generation in dyadic conversation. It contributes ListenerX, a new dataset of roughly 1.4M frames of 3D dyadic conversations with FLAME parameters, fine-grained textual descriptions (generated with an action-unit detector plus a vision-language model and then manually revised), and continuous valence/arousal emotion-intensity tags. The proposed VividListener system is a diffusion-transformer model conditioned on speaker motion, speaker audio, listener text descriptions, and emotional intensity tags; it includes a Responsive Interaction Module for multimodal fusion and an Emotional Intensity Tagging mechanism for intensity modulation. Experiments on ListenerX and on a cross-scenario DailyX-to-InterviewX setting compare against L2L, RLHG, and DIM, with ablations and a user study, and report state-of-the-art results.

Significance. If the controllability claims were independently verified, the paper would make a solid contribution: ListenerX appears to be the largest long-sequence listener-dynamics dataset with fine-grained text and intensity annotations, and the proposed diffusion-transformer design with adaptive multimodal fusion is reasonable and clearly better than the retrained baselines on the reported metrics. The dataset alone could be valuable to the community. However, the central claim of fine-grained, semantically controllable listener generation is currently supported only by evidence that is partly circular: the conditioning signals are derived from the same ground-truth listener motions used for both training and evaluation, and the quantitative metrics compare against those same targets. Unless the authors add independent controllability evaluations and annotation-quality validation, the headline claim remains plausible but not established.

major comments (4)
  1. [Sec. 3.2, Sec. 5.1, Table 2] The conditioning and evaluation loop is circular. The text descriptions are produced from the listener's own facial frames via an AU detector plus VLM, then manually revised, and the intensity tags are computed by an automatic V/A estimator applied to the same frames (Sec. 3.2). All main quantitative metrics (FD, P-FD, MSE) compare generated motion to that same ground-truth motion (Table 2). A model that receives target-derived conditions can therefore improve reconstruction over baselines that do not receive them, without showing that a user-specified open-vocabulary description or a requested intensity level is actually followed. I request independent tests of controllability: (i) a condition-swap experiment in which text or intensity tags are changed while speaker input is fixed, with a check that the output changes in the intended direction; (ii) a text-to-motion alignment metric computed on held-out conditions that are not derived from the target listener motion; and (iii) an intensity-scaling test in which the tag is varied and the predicted V/A of the generated output is measured. Without such tests, the fine-grained control claim is not supported by the current experiments.
  2. [Sec. 4.3, Eq. (5), Table 3] The emotional intensity loss in Eq. (5) uses a pre-trained predictor P_emotional on the generated motion. If this predictor is the same estimator used in Sec. 3.2 to create the intensity tags, then the model is trained to reproduce the estimator's own biases, and the quantitative intensity-control evidence becomes self-referential. The paper never identifies P_emotional or states whether it is identical to the annotation model [50], so this needs to be clarified. If the same model is used, the authors should provide human validation of intensity alignment or use an independent predictor. Relatedly, Table 3 refers to an 'Emotional Identity Predictor (EIP)' while Sec. 4.3 and Eq. (5) describe an 'Emotional Intensity Predictor'; this naming inconsistency should be fixed.
  3. [Sec. 5.2, Table 2] The headline cross-scenario claim is not accurately quantified or statistically supported. The text says the method achieves a 'reduction in FD by 25.21%', but 41.510 - 16.309 = 25.201 is an absolute FD difference, not a percentage reduction (the relative reduction is about 60.7%). Also, no error bars, confidence intervals, or significance tests are provided for any metric, and the cross-scenario experiment uses a single train/test split. Given that the conditions include text and intensity tags extracted from the test-set listener motions, the comparison is potentially unfair to the baselines, which do not receive those target-derived signals. The authors should report variability across seeds, run a significance test, and ideally include a baseline that also receives the same text/tag inputs, to isolate the benefit of the proposed architecture from the benefit of target-derived conditioning information.
  4. [Sec. 5.2, Sec. 5.3, Table 4, Fig. 5] The ablations and user study do not test semantic alignment with the input text or intensity levels. Table 4 shows that removing text or tags degrades FD, SID, Var, and rPCC, but those metrics measure reconstruction quality or diversity, not whether the model uses the text semantically or scales intensity appropriately. A model could overfit to arbitrary condition codes and still show such degradation. The user study in Fig. 5 asks raters to judge similarity, interactivity, and diversity; none of the questions asks whether the generated motion matches the supplied text description or the requested emotional intensity. I recommend adding a human evaluation with forced-choice questions such as 'which video better matches the text description' and 'which video has stronger emotional intensity,' and reporting an automated alignment metric (e.g., CLIP-based similarity between the description and rendered generated frames).
minor comments (6)
  1. [Sec. 2.2, References] In the related work, the citation '38 ?' appears in an incomplete form; the reference list should be cleaned up.
  2. [Table 1] The header row 'CV P R' and entries such as 'ListenerX(Ours 2025)' are unclear or inconsistently formatted; the table would benefit from a caption defining abbreviations and a cleaner layout.
  3. [Eq. (2)] Please define the transposition notation E_text' explicitly and state the dimensions of W_fuse, E_fused, and E_text; currently the Hadamard product between E_fused and the pooled weight vector is under-specified.
  4. [Sec. 5.3 and Fig. 4/Fig. 5] The cross-references are inconsistent: Sec. 5.3 states that Fig. 5 illustrates conditional control results, but Fig. 5 is the user-study figure, whereas Fig. 4 shows conditional control examples. Please correct the references.
  5. [Sec. 5.2, Table 2] The phrase 'sub-optimal counterparts' should identify which baseline is considered sub-optimal and define how the reduction percentage is computed.
  6. [Sec. 3.2] The term 'authoritative emotional intensity tags' is overstated, since the tags come from an automatic single-model estimator; the paper should acknowledge this and provide some validation, such as correlation with human V/A ratings or inter-annotator agreement.

Circularity Check

2 steps flagged · score 5.0 of 10

Quantitative controllability evidence is self-consistent: text and intensity conditions are extracted from the target listener motions, and no independent text-alignment or intensity-control metric is reported.

  1. self definitional [Section 3.2 (Multi-Model Annotation Pipeline) and Eq. (4)]
    "we incorporate the facial action unit detector [4] to extract high-intensity action units that serve as additional prompts for VLM. In this paradigm, we obtain accurate textual descriptions of the listener expressions."

    The text descriptions that condition the generation are computed from the same ground-truth listener frames that the model is trained to reconstruct. Equation (4) minimizes ||H - VividL(H(t), Sm, Sa, text, sigma, t)||^2, where H is the target listener motion and text (and the intensity tags) are derived from that same H in Section 3.2. Therefore the conditioning inputs are functions of the output by construction: text = f(H), sigma = g(H). A model trained to invert these functions will trivially score better than baselines that do not receive target-derived annotations.

  2. fitted input called prediction [Section 4.3, Eq. (5) and Section 5.3 User Study]
    "we employ a pre-trained 3D emotional intensity predictor to produce the corresponding emotion tag for our results. The emotional intensity loss is defined as follows: Lemotional =∥σ−P emotional( ˆH)∥2 2, (5)"

    The sigma in Eq. (5) is the valence/arousal tag produced by the facial affect analysis [50] applied to the ground-truth frames in Section 3.2. The loss therefore trains the generated motion to match the tag-generation estimator's own output, making the 'emotional intensity control' evidence self-referential: the target is defined by the estimator and the success is measured by the same estimator. The user study, which the paper claims gives 'strong user affirmation for its alignment with input conditions,' actually asks participants to rate only 'similarity to GT, interactivity, and diversity' — not whether the output matches the input text or intensity tags.

full rationale

VividListener introduces a new dataset and a diffusion-transformer framework; the architecture itself is not derived from its inputs. However, the paper's quantitative case for fine-grained text- and intensity-based control is partially circular. The text descriptions and intensity tags that serve as conditioning signals are extracted from the same ground-truth listener motions that the model is trained to reconstruct, and the main metrics (FD, P-FD, MSE) compare against that same ground truth. This gives VividListener an information advantage over the baselines, which do not receive target-derived conditions, and the ablations in Table 4 only show that removing this target-derived information hurts reconstruction — expected behavior for a model inverting its own annotation functions. The emotional intensity loss additionally reinforces consistency with the V/A estimator used to create the tags, and no independent metric for semantic alignment or intensity scaling is reported. No load-bearing self-citation was found: the cited AU detector, VLM, and affect-analysis tools are external, and the authors' own prior works are not used to justify the core architecture. The paper is not self-contained against external benchmarks, as it evaluates only on its own dataset, so the comparison is not an independent test of controllability. Because the generative model has independent architectural content and the qualitative open-vocabulary examples (Fig. 4) suggest some generalization, the circularity is partial rather than complete. Score 5 reflects a central claim whose quantitative support is partly self-consistent by construction, without reducing to a fully fitted tautology.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The method relies on standard pretrained components (EMOCA, CLIP, V/A estimator, diffusion) and on the quality of the automatically generated annotations. The only hand-tuned free parameters are the loss weights. No new physical entities are introduced.

free parameters (1)
  • Loss weights lambda_simple, lambda_emotional, lambda_vel = 2.0, 0.2, 0.8
    Empirically set in training (Section 5.1); the intensity control claim depends on lambda_emotional balancing the pre-trained predictor loss.
assumptions (4)
  • domain assumption EMOCA provides accurate FLAME fits for in-the-wild dyadic videos.
    Used in Section 3.2 for 3D reconstruction; if fits are noisy, all downstream motion quality claims degrade.
  • domain assumption CLIP text embeddings capture semantics of expression descriptions.
    Used in RIM; the semantic alignment between text and motion relies on CLIP's quality.
  • domain assumption Valence/Arousal estimator [50] yields valid emotion intensity tags.
    Used to create sigma tags and to train the EIP; if V/A estimates are biased, the intensity control claims weaken.
  • domain assumption The pre-trained emotional intensity predictor P_emotional generalizes to generated motions.
    Used in the loss of Eq. 5; if it does not generalize, the intensity consistency loss is ineffective.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VividListener: Expressive and Controllable Listener Dynamics Modeling for Multi-Modal Responsive Interaction." pith.science (2026). https://pith.science/paper/XDDPOBZH

@misc{pith2026250421718,
  author       = {Pith},
  title        = {Pith review of: VividListener: Expressive and Controllable Listener Dynamics Modeling for Multi-Modal Responsive Interaction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XDDPOBZH}},
  note         = {Machine review of arXiv:2504.21718}
}
read the original abstract

Generating responsive listener head dynamics with nuanced emotions and expressive reactions is crucial for practical dialogue modeling in various virtual avatar animations. Previous studies mainly focus on the direct short-term production of listener behavior. They overlook the fine-grained control over motion variations and emotional intensity, especially in long-sequence modeling. Moreover, the lack of long-term and large-scale paired speaker-listener corpora including head dynamics and fine-grained multi-modality annotations (e.g., text-based expression descriptions, emotional intensity) also limits the application of dialogue modeling.Therefore, we first newly collect a large-scale multi-turn dataset of 3D dyadic conversation containing more than 1.4M valid frames for multi-modal responsive interaction, dubbed ListenerX. Additionally, we propose VividListener, a novel framework enabling fine-grained, expressive and controllable listener dynamics modeling. This framework leverages multi-modal conditions as guiding principles for fostering coherent interactions between speakers and listeners.Specifically, we design the Responsive Interaction Module (RIM) to adaptively represent the multi-modal interactive embeddings. RIM ensures the listener dynamics achieve fine-grained semantic coordination with textual descriptions and adjustments, while preserving expressive reaction with speaker behavior. Meanwhile, we design the Emotional Intensity Tags (EIT) for emotion intensity editing with multi-modal information integration, applying to both text descriptions and listener motion amplitude.Extensive experiments conducted on our newly collected ListenerX dataset demonstrate that VividListener achieves state-of-the-art performance, realizing expressive and controllable listener dynamics.

Figures

Figures reproduced from arXiv: 2504.21718 by the authors.

Figure 1
Figure 1. VividListener: listener dynamics modeling framework for multi-modal responsive interaction. This framework inputs speaker [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. VividListener: listener dynamics modeling framework for multi-modal responsive interaction. This framework first integrates [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Visual comparisons on ListenerX. We present visu￾alizations of listener motions generated on ListenerX compared to various state-of-the-art methods. Unlike other approaches, our VividListener input incorporates fine-grained textual descriptions (shown in the last row). Where ϵ ∼ N (0, I) is the added random Gaussian noise, H(t) = H +γ(t)ϵ represents the gradual noise addition pro￾cess at step t. γ(t) ∈ (0, 1) is a c… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Conditional Control Results. We generate listener head dynamics by inputting the speaker audio and head motion information (shown in the top-left corner) alongside dataset-based emotion descriptions (shown in the bottom-left corner). Additionally, we utilize open-vocab…
Figure 5
Figure 5. Figure 5: User Study. Comparisons with SOTA methods. surpasses the limitations of fixed-text descriptions in tra￾ditional datasets. For instance, Description 1 captures a gradual transition from a ”somber expression” to a ”bright￾ened mood,” while Description 2 showcases richer …
Figure 6
Figure 6. Figure 6: Visual comparisons on cross-scenario. We present visualizations of listener motions generated in a cross-scenario setting, where all models are trained on the DailyX subset and tested on the InterviewX dataset. Speaker : (Person on the Right) Listener: (Person on the L…
Figure 7
Figure 7. Figure 7: Multi-turn Interaction Result. We generated the results of a dyadic multi-turn conversation, where the roles of the speaker and listener dynamically alternate. (In the first row) the person on the left speaks, while the person on the right listens. (In the second row) …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

60 extracted references · 42 canonical work pages

  1. [50]

    Estimation of continuous va- lence and arousal levels from faces in naturalistic conditions

    Antoine Toisoul, Jean Kossaifi, Adrian Bulat, Georgios Tz- imiropoulos, and Maja Pantic. Estimation of continuous va- lence and arousal levels from faces in naturalistic conditions. Nature Machine Intelligence, 3(1):42–50, 2021. 2, 4

    work page 2021

  2. [1]

    Facetalk: Audio-driven motion diffusion for neural parametric head models

    Shivangi Aneja, Justus Thies, Angela Dai, and Matthias Nießner. Facetalk: Audio-driven motion diffusion for neural parametric head models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 21263–21273, 2024. 3

    work page 2024

  3. [2]

    On the challenges and opportu- nities of physically situated dialog

    Dan Bohus and Eric Horvitz. On the challenges and opportu- nities of physically situated dialog. In 2010 AAAI Fall Sym- posium Series, 2010. 3

    work page 2010

  4. [3]

    Facilitating multiparty dia- log with gaze, gesture, and speech

    Dan Bohus and Eric Horvitz. Facilitating multiparty dia- log with gaze, gesture, and speech. In International Con- ference on Multimodal Interfaces and the Workshop on Ma- chine Learning for Multimodal Interaction, pages 1–8, 2010. 3

    work page 2010

  5. [4]

    Libreface: An open-source toolkit for deep facial expression analysis

    Di Chang, Yufeng Yin, Zongjian Li, Minh Tran, and Mo- hammad Soleymani. Libreface: An open-source toolkit for deep facial expression analysis. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 8205–8215, 2024. 2, 4

    work page 2024

  6. [5]

    Us- ability and responsiveness of artificial intelligence chatbot on online customer experience in e-retailing

    Ja-Shen Chen, Le Tran-Thien-Y , and Devina Florence. Us- ability and responsiveness of artificial intelligence chatbot on online customer experience in e-retailing. International Journal of Retail & Distribution Management, 49(11):1512– 1531, 2021. 2

    work page 2021

  7. [6]

    J. S. Chung and A. Zisserman. Out of time: automated lip sync in the wild. In Workshop on Multi-view Lip-reading, ACCV, 2016. 4

    work page 2016

  8. [7]

    Out of time: auto- mated lip sync in the wild

    Joon Son Chung and Andrew Zisserman. Out of time: auto- mated lip sync in the wild. In Computer Vision–ACCV 2016 Workshops: ACCV 2016 International Workshops, Taipei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part II 13, pages 251–263. Springer, 2017

    work page 2016

Show all 60 references
  1. [8]

    Capture, learning, and synthe- sis of 3d speaking styles

    Daniel Cudeiro, Timo Bolkart, Cassidy Laidlaw, Anurag Ranjan, and Michael J Black. Capture, learning, and synthe- sis of 3d speaking styles. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10101–10111, 2019. 3

    2019

  2. [9]

    Emoca: Emotion driven monocular face capture and animation

    Radek Dan ˇeˇcek, Michael J Black, and Timo Bolkart. Emoca: Emotion driven monocular face capture and animation. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 20311–20322, 2022. 2, 4

    2022

  3. [10]

    Emotional speech-driven animation with content-emotion disentangle- ment

    Radek Dan ˇeˇcek, Kiran Chhatre, Shashank Tripathi, Yan- dong Wen, Michael Black, and Timo Bolkart. Emotional speech-driven animation with content-emotion disentangle- ment. In SIGGRAPH Asia 2023 Conference Papers , pages 1–13, 2023. 3

    2023

  4. [11]

    A survey of embodied ai: From simulators to research tasks

    Jiafei Duan, Samson Yu, Hui Li Tan, Hongyuan Zhu, and Cheston Tan. A survey of embodied ai: From simulators to research tasks. IEEE Transactions on Emerging Topics in Computational Intelligence, 6(2):230–244, 2022. 2

    2022

  5. [12]

    Facial action coding system

    Paul Ekman and Wallace V Friesen. Facial action coding system. Environmental Psychology & Nonverbal Behavior, 1978

    1978

  6. [13]

    Studying human robot interaction and its characteristics

    Maged Farouk. Studying human robot interaction and its characteristics. International Journal of Computations, In- formation and Manufacturing (IJCIM), 2(1), 2022

    2022

  7. [14]

    for real

    Ad ´ela¨ıde Genay, Anatole L ´ecuyer, and Martin Hachet. Be- ing an avatar “for real”: a survey on virtual embodiment in augmented reality. IEEE Transactions on Visualization and Computer Graphics, 28(12):5071–5090, 2021. 2

    2021

  8. [15]

    Affective faces for goal-driven dyadic communication

    Scott Geng, Revant Teotia, Purva Tendulkar, Sachit Menon, and Carl V ondrick. Affective faces for goal-driven dyadic communication. arXiv preprint arXiv:2301.10939, 2023. 4

    2023 arXiv

  9. [16]

    Gans trained by a two time-scale update rule converge to a local nash equilib- rium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium. Advances in neural information processing systems , 30, 2017. 12

    2017

  10. [17]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020

    2020

  11. [18]

    Arbitrary style transfer in real-time with adaptive instance normalization

    Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceed- ings of the IEEE international conference on computer vi- sion, pages 1501–1510, 2017. 6

    2017

  12. [19]

    Dyadgan: Generating fa- cial expressions in dyadic interactions

    Yuchi Huang and Saad M Khan. Dyadgan: Generating fa- cial expressions in dyadic interactions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni- tion Workshops, pages 11–18, 2017. 3

    2017

  13. [20]

    Gradient-based learning applied to document recog- nition

    Yann LeCun, L ´eon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recog- nition. Proceedings of the IEEE , 86(11):2278–2324, 1998. 6

    1998

  14. [21]

    Learning a model of facial shape and expression from 4d scans

    Tianye Li, Timo Bolkart, Michael J Black, Hao Li, and Javier Romero. Learning a model of facial shape and expression from 4d scans. ACM Trans. Graph., 36(6):194–1, 2017. 2, 5

    2017

  15. [22]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 2, 4

    2024

  16. [23]

    Mfr-net: Multi-faceted responsive listening head generation via denoising diffusion model

    Jin Liu, Xi Wang, Xiaomeng Fu, Yesheng Chai, Cai Yu, Jiao Dai, and Jizhong Han. Mfr-net: Multi-faceted responsive listening head generation via denoising diffusion model. In Proceedings of the 31st ACM International Conference on Multimedia, pages 6734–6743, 2023. 2, 3

    2023

  17. [24]

    Customlistener: Text-guided responsive inter- action for user-friendly listening head generation

    Xi Liu, Ying Guo, Cheng Zhen, Tong Li, Yingying Ao, and Pengfei Yan. Customlistener: Text-guided responsive inter- action for user-friendly listening head generation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2415–2424, 2024. 2, 3

    2024

  18. [25]

    Decoupled weight decay regularization

    I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 7, 12

    2017 arXiv

  19. [26]

    Automated measurement of facial ex- pression in infant–mother interaction: A pilot study.Infancy, 14(3):285–305, 2009

    Daniel S Messinger, Mohammad H Mahoor, Sy-Miin Chow, and Jeffrey F Cohn. Automated measurement of facial ex- pression in infant–mother interaction: A pilot study.Infancy, 14(3):285–305, 2009. 12

    2009

  20. [27]

    Learning to listen: Modeling non-deterministic dyadic facial motion

    Evonne Ng, Hanbyul Joo, Liwen Hu, Hao Li, Trevor Darrell, Angjoo Kanazawa, and Shiry Ginosar. Learning to listen: Modeling non-deterministic dyadic facial motion. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20395–20405, 2022. 2...

    2022

  21. [28]

    Can language models learn to listen? In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision , pages 10083– 10093, 2023

    Evonne Ng, Sanjay Subramanian, Dan Klein, Angjoo Kanazawa, Trevor Darrell, and Shiry Ginosar. Can language models learn to listen? In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision , pages 10083– 10093, 2023. 3

    2023

  22. [29]

    Improved denoising diffusion probabilistic models

    Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International conference on machine learning, pages 8162–8171. PMLR,

  23. [30]

    Interactive generative adversarial networks for facial ex- pression generation in dyadic interactions

    Behnaz Nojavanasghari, Yuchi Huang, and Saad Khan. Interactive generative adversarial networks for facial ex- pression generation in dyadic interactions. arXiv preprint arXiv:1801.09092, 2018. 3

    2018 arXiv

  24. [31]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 4195–4205,

  25. [32]

    Deepfake generation and detection: A benchmark and survey

    Gan Pei, Jiangning Zhang, Menghan Hu, Zhenyu Zhang, Chengjie Wang, Yunsheng Wu, Guangtao Zhai, Jian Yang, Chunhua Shen, and Dacheng Tao. Deepfake generation and detection: A benchmark and survey. arXiv preprint arXiv:2403.17881, 2024. 2

    2024

  26. [33]

    Emotalk: Speech-driven emotional disentanglement for 3d face anima- tion

    Ziqiao Peng, Haoyu Wu, Zhenbo Song, Hao Xu, Xiangyu Zhu, Jun He, Hongyan Liu, and Zhaoxin Fan. Emotalk: Speech-driven emotional disentanglement for 3d face anima- tion. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pages 20687–20697, 2023. 3

    2023

  27. [34]

    Emotiongesture: Audio-driven diverse emo- tional co-speech 3d gesture generation

    Xingqun Qi, Chen Liu, Lincheng Li, Jie Hou, Haoran Xin, and Xin Yu. Emotiongesture: Audio-driven diverse emo- tional co-speech 3d gesture generation. IEEE Transactions on Multimedia, 2024. 2

    2024

  28. [35]

    Weakly-supervised emotion transition learning for diverse 3d co-speech gesture generation

    Xingqun Qi, Jiahao Pan, Peng Li, Ruibin Yuan, Xiaowei Chi, Mengfei Li, Wenhan Luo, Wei Xue, Shanghang Zhang, Qifeng Liu, et al. Weakly-supervised emotion transition learning for diverse 3d co-speech gesture generation. InPro- ceedings of the IEEE/CVF Conference on Computer Vis...

    2024

  29. [36]

    Cocogesture: Toward coherent co- speech 3d gesture generation in the wild

    Xingqun Qi, Hengyuan Zhang, Yatian Wang, Jiahao Pan, Chen Liu, Peng Li, Xiaowei Chi, Mengfei Li, Qixun Zhang, Wei Xue, et al. Cocogesture: Toward coherent co- speech 3d gesture generation in the wild. arXiv preprint arXiv:2405.16874, 2024. 2

    2024 arXiv

  30. [37]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  31. [38]

    Meshtalk: 3d face an- imation from speech using cross-modality disentanglement

    Alexander Richard, Michael Zollh ¨ofer, Yandong Wen, Fer- nando De la Torre, and Yaser Sheikh. Meshtalk: 3d face an- imation from speech using cross-modality disentanglement. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1173–1182, 2021. 3

    2021

  32. [39]

    Quantifying facial expression synchrony in face-to- face dyadic interactions: Temporal dynamics of simultane- ously recorded facial emg signals

    Marcel Riehle, J ¨urgen Kempkensteffen, and Tania M Lin- coln. Quantifying facial expression synchrony in face-to- face dyadic interactions: Temporal dynamics of simultane- ously recorded facial emg signals. Journal of Nonverbal Be- havior, 41:85–102, 2017. 12

    2017

  33. [40]

    A circumplex model of affect

    James A Russell. A circumplex model of affect. Journal of personality and social psychology, 39(6):1161, 1980. 4

    1980

  34. [41]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020. 12

    2010 arXiv

  35. [42]

    Emotional listener portrait: Neural lis- tener head generation with emotion

    Luchuan Song, Guojun Yin, Zhenchao Jin, Xiaoyi Dong, and Chenliang Xu. Emotional listener portrait: Neural lis- tener head generation with emotion. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 20839–20849, 2023. 2, 3

    2023

  36. [43]

    A con- versational agent framework with multi-modal personality expression

    Sinan Sonlu, U ˘gur G¨ud¨ukbay, and Funda Durupinar. A con- versational agent framework with multi-modal personality expression. ACM Transactions on Graphics (TOG) , 40(1): 1–16, 2021. 3

    2021

  37. [44]

    Artificial intelligence, machine learning and deep learning in advanced robotics, a review

    Mohsen Soori, Behrooz Arezoo, and Roza Dastres. Artificial intelligence, machine learning and deep learning in advanced robotics, a review. Cognitive Robotics, 3:54–70, 2023. 2

    2023

  38. [45]

    Analyzing human– human interactions: A survey

    Alexandros Stergiou and Ronald Poppe. Analyzing human– human interactions: A survey. Computer Vision and Image Understanding, 188:102799, 2019. 2

    2019

  39. [46]

    Avi-talking: Learning audio-visual in- structions for expressive 3d talking face generation

    Yasheng Sun, Wenqing Chu, Hang Zhou, Kaisiyuan Wang, and Hideki Koike. Avi-talking: Learning audio-visual in- structions for expressive 3d talking face generation. IEEE Access, 2024

    2024

  40. [47]

    Diffposetalk: Speech-driven stylistic 3d facial animation and head pose generation via diffusion models

    Zhiyao Sun, Tian Lv, Sheng Ye, Matthieu Lin, Jenny Sheng, Yu-Hui Wen, Minjing Yu, and Yong-jin Liu. Diffposetalk: Speech-driven stylistic 3d facial animation and head pose generation via diffusion models. ACM Transactions on Graphics (TOG), 43(4):1–9, 2024. 3

    2024

  41. [48]

    Listen- ing head motion generation for multimodal dialog system

    Mikawa Tamon, Fujii Yasuhisa, Wakabayashi Yukoh, Ohta Kengo, Nishimura Ryota, and Kitaoka Norihide. Listen- ing head motion generation for multimodal dialog system. In 2024 11th International Conference on Advanced Infor- matics: Concept, Theory and Application (ICAI), pages 1...

    2024

  42. [49]

    Imitator: Personalized speech-driven 3d facial animation

    Balamurugan Thambiraja, Ikhsanul Habibie, Sadegh Aliak- barian, Darren Cosker, Christian Theobalt, and Justus Thies. Imitator: Personalized speech-driven 3d facial animation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 20621–20631, 2023. 3

    2023

  43. [51]

    Dim: Dyadic interaction modeling for social be- havior generation

    Minh Tran, Di Chang, Maksim Siniukov, and Mohammad Soleymani. Dim: Dyadic interaction modeling for social be- havior generation. In European Conference on Computer Vision, pages 484–503. Springer, 2024. 2, 3, 7, 8

    2024

  44. [52]

    Dyadic interaction modeling for social behavior generation

    Minh Tran, Di Chang, Maksim Siniukov, and Mohammad Soleymani. Dyadic interaction modeling for social behavior generation. arXiv preprint arXiv:2403.09069, 2024

    2024 arXiv

  45. [53]

    Neural discrete representation learning

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information pro- cessing systems, 30, 2017

    2017

  46. [54]

    Effects of inter- acting with a crowd of emotional virtual humans on users’ affective and non-verbal behaviors

    Matias V olonte, Yu-Chun Hsu, Kuan-Yu Liu, Joe P Mazer, Sai-Keung Wong, and Sabarish V Babu. Effects of inter- acting with a crowd of emotional virtual humans on users’ affective and non-verbal behaviors. In 2020 IEEE Confer- ence on Virtual Reality and 3D User Interfaces (VR)...

    2020

  47. [55]

    Versa- tile face animator: Driving arbitrary 3d facial avatar in rgbd space

    Haoyu Wang, Haozhe Wu, Junliang Xing, and Jia Jia. Versa- tile face animator: Driving arbitrary 3d facial avatar in rgbd space. In Proceedings of the 31st ACM International Con- ference on Multimedia, pages 7776–7784, 2023

    2023

  48. [56]

    Speech-driven 3d face animation with com- posite and regional facial movements

    Haozhe Wu, Songtao Zhou, Jia Jia, Junliang Xing, Qi Wen, and Xiang Wen. Speech-driven 3d face animation with com- posite and regional facial movements. In Proceedings of the 31st ACM International Conference on Multimedia , pages 6822–6830, 2023. 3

    2023

  49. [57]

    Motiondif- fuse: Text-driven human motion generation with diffusion model

    Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. Motiondif- fuse: Text-driven human motion generation with diffusion model. arXiv preprint arXiv:2208.15001, 2022. 6

    2022 arXiv

  50. [58]

    Media2face: Co-speech facial animation gen- eration with multi-modality guidance

    Qingcheng Zhao, Pengyu Long, Qixuan Zhang, Dafei Qin, Han Liang, Longwen Zhang, Yingliang Zhang, Jingyi Yu, and Lan Xu. Media2face: Co-speech facial animation gen- eration with multi-modality guidance. In ACM SIGGRAPH 2024 conference papers, pages 1–13, 2024. 3

    2024

  51. [59]

    Responsive listening head generation: a benchmark dataset and baseline

    Mohan Zhou, Yalong Bai, Wei Zhang, Ting Yao, Tiejun Zhao, and Tao Mei. Responsive listening head generation: a benchmark dataset and baseline. In European Conference on Computer Vision, pages 124–142. Springer, 2022. 2, 3, 4, 7, 8

    2022

  52. [60]

    To your family and nation, being a trickster is destructive, but it’s like finding God in freedom

    Supplementary material 7.1. Implementation Details In our experiments, we set the total generated sequence length N = 240 with the normalized fps=30. Thus, the du- ration of each sequence is 8 seconds. Sa is initially rep- resented as audio signal waves, which are converted in...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.