Pith. sign in

REVIEW 2 major objections 4 minor 78 references

From one text prompt, CHAT generates paired, mutually responsive audio-visual dialogues that can stand in for scarce real dyadic data.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 06:58 UTC pith:P5LYMYP6

load-bearing objection Solid systems paper that properly defines single-prompt DIADG and ships a working modular pipeline plus a large synthetic set; evaluation is the soft part, and the authors mostly own it. the 2 major comments →

arxiv 2607.02799 v1 pith:P5LYMYP6 submitted 2026-07-02 cs.CV

Conversational Human Audio-visual Talking Dialogue Generation

Junhao Song , Lluis Guasch , Xilin He , Zhongyu Yang , Yingfang Yuan , Weicheng Xie , Linlin Shen , Haijun Lin
show 3 more authors
Shizhe Liu Wei Pang Siyang Song
This is my paper
classification cs.CV
keywords dyadic interactive audio-visual dialogueDIADGtalking face generationfacial reaction generationinteractive audio refinementdiffusion modelsynthetic dialogue datasetREACT 2024
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Real dyadic interactive audio-visual dialogue recordings are expensive, ethically constrained, and demographically limited, yet they are the data that virtual agents and digital humans need. This paper defines the DIADG task and presents CHAT, a modular pipeline that turns a single anonymous scenario prompt into many diverse, paired speech-and-face dialogue clips. Large language models first write multi-turn scripts and identity descriptions; an interactive audio refinement stage then produces emotion-aware, turn-taking speech with occasional listener backchannels; an interactive facial behaviour refinement stage finally turns those audio tracks into temporally coherent face videos whose silent and speaking segments respond to the partner. The resulting clips beat adapted talking-face and facial-reaction baselines on synchronisation, interactivity, identity consistency, and a 60-person user study. The same system yields the 50k-pair CHAT-AVD-50k corpus, which, used as pre-training, improves two interactive-head models on the REACT 2024 benchmark. The practical claim is that synthetic, prompt-driven DIAD data can scale coverage without further real recording.

Core claim

A single textual scenario prompt is sufficient, once routed through textual dialogue generation, interactive audio refinement, and interactive facial behaviour refinement, to produce diverse, temporally coherent, and mutually responsive dyadic audio-visual dialogue pairs that match or exceed related methods on synchronisation, reaction quality, identity preservation, and human preference, and that supply effective pre-training data for interactive head generation.

What carries the argument

The Interactive Facial Behaviour Refinement (IFBR) block (with Silent Facial Behaviour Generation, Responsive Facial Behaviour Generation that injects multi-scale partner audio-visual conditions into a diffusion transformer, and Temporal Continuity Refinement), working together with Interactive Audio Refinement, converts non-interactive speech and face segments into mutually responsive DIAD clips.

Load-bearing premise

The paper treats short-clip audio-visual sync scores and facial-reaction appropriateness metrics, applied to the first ten seconds and to baselines taken from related but different tasks, as adequate evidence of full dialogue-level verbal and non-verbal mutual responsiveness.

What would settle it

Train the same interactive-head models on an equal volume of real multi-turn dyadic data versus CHAT-AVD-50k, then measure whether the synthetic pre-training still improves FRCorr/FRDist and human interactivity ratings on a held-out real corpus whose distribution is independent of REACT 2024.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Large-scale, demographically varied DIAD training sets can be expanded from text prompts without new human recording sessions.
  • Interactive head-generation models gain measurable reaction appropriateness and lower distance to ground-truth reactions after CHAT-AVD-50k pre-training.
  • The modular pipeline lets individual LLM, TTS, or talking-face components be swapped without retraining the whole system.
  • Synthetic DIAD clips become a practical alternative when privacy or ethics bar real dyadic capture.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Dialogue-level metrics that score turn-taking timing and semantic coherence, not only per-clip facial reaction scores, would be needed before claiming full conversational fidelity.
  • The same refinement blocks could be applied to multi-party or multilingual prompts once the underlying LLMs and TTS models support those languages.
  • Downstream personalisation of virtual agents could treat CHAT-generated identity-and-emotion pairs as a controllable prior rather than only as pre-training bulk.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper defines Dyadic Interactive Audio-visual Dialogue Generation (DIADG): from a single anonymous textual scenario prompt, produce multiple diverse pairs of mutually responsive speech-and-face clips. CHAT implements this with three modules—TDG (LLM dialogue and identity text), DADG with Interactive Audio Refinement (IAR: interactive words, emotion/prosody descriptors, timestamps, sound environment, then emotion-aware TTS), and IFBG with Interactive Facial Behaviour Refinement (IFBR: SilentDiff for silent segments, multi-scale RFBG diffusion conditioned on the partner’s audio-visual behaviour, and Gaussian TCR boundary blending). The authors release CHAT-AVD-50k (50k pairs, ~1389 h) and report that CHAT beats adapted talking-face and facial-reaction baselines on FID/FVD, LSE-C/D, FRCorr/FRDiv, CSIM/LPIPS and a 60-person user study, while pre-training PerFRDiff and ReactDiff on CHAT-AVD-50k improves REACT 2024 metrics.

Significance. If the claims hold, the work supplies both a clean task definition for full verbal+non-verbal dyadic synthesis from text alone and a practical modular pipeline that can scale demographically diverse DIAD data without real multi-person capture. The ablations (Tables 3–4) cleanly attribute gains to IAR emotion conditioning and RFBG partner conditioning; the user study is large and multi-region; and the modular design (frozen LLMs/TTS, trainable SFBG/RFBG) is engineering-sensible. These are real contributions for digital humans and HCI data. The main caveats are evaluation scope (first 10 s, FRG metrics that the paper itself says miss dialogue-level turn-taking) and the related-distribution nature of the downstream pre-training claim, which the authors already flag in Sec. 5.5.

major comments (2)
  1. Sec. 5.5 and Table 5: The claim that CHAT-AVD-50k is “effective pre-training data” for interactive head generation is load-bearing for the abstract and conclusion, yet IFBR is itself pre-trained on REACT 2024 (Sec. 5.1). The paper correctly calls the set “a large-scale diverse augmentation over a related interaction distribution rather than as a fully out-of-domain source.” The reported FRCorr/FRDist gains therefore cannot support independent transfer or a clean alternative to real DIAD collection. Either add an evaluation on an independent corpus (e.g., IEMOCAP/NoXi as the authors themselves list in Sec. 7) or rephrase the abstract/conclusion to match the related-distribution interpretation already stated in Sec. 5.5.
  2. Sec. 5.2 and Tables 1–2: No baseline solves DIADG; comparisons use talking-face and FRG methods adapted with CHAT-generated audio/identity or speaker video, restricted to the first ten seconds. FRCorr/FRDiv are facial-reaction metrics that the paper notes “do not fully reflect turn-taking or semantic coherence at the dialogue level.” Concurrent joint systems (AV-Flow, DualTalk, INFP, ChatAnyone, TAVID, JAM-Flow) are excluded for lack of code. The outperformance claim is therefore only relative to related-task proxies. Strengthen by (i) reporting full-clip or multi-turn metrics (turn-taking latency, backchannel rate, dialogue-level semantic coherence) and (ii) at least qualitative or partial comparison to one concurrent joint-generation system if any public demo/API exists.
minor comments (4)
  1. Eq. (4) and free parameters: P_inter, W, σ=W/3, L and l(t)=max(1,⌈L t/T⌉), and the 5–10 turn constraint are free design choices. A short sensitivity note (or fixed defaults with justification) would help reproducibility.
  2. Notation: bar/hat/tilde conventions are introduced once but mixed with #V and /V in Fig. 2; a compact symbol table would reduce ambiguity.
  3. Sec. 7 already lists English-only scope and REACT dependence; ensure the abstract does not over-claim “scalable alternative” without those caveats.
  4. Fig. 4 qualitative comparison is useful; adding a failure case (e.g., long multi-turn drift or identity leakage) would balance the presentation.

Circularity Check

1 steps flagged

Main DIADG claims rest on external metrics and a user study; only the secondary CHAT-AVD-50k pre-training claim has a mild related-distribution self-loop that the paper itself flags.

specific steps
  1. other [Sec. 5.5 / Table 5 (and Sec. 5.1 pre-training note)]
    "Since CHAT's IFBR block is itself pre-trained on REACT 2024 (Sec. 5), CHAT-AVD-50k should be regarded as a large-scale diverse augmentation over a related interaction distribution rather than as a fully out-of-domain source. ... We therefore interpret these gains as evidence of useful scale and diversity augmentation within a related distribution, rather than as strictly independent transfer."

    IFBR is trained on REACT 2024; CHAT-AVD-50k is generated with that IFBR; PerFRDiff and ReactDiff (same author group as REACT) are then pre-trained on CHAT-AVD-50k and fine-tuned/evaluated on REACT. Gains (FRCorr/FRDist) therefore measure in-distribution scale/diversity augmentation, not independent transfer. The paper states this hedge, so the abstract claim that CHAT-AVD-50k is “effective pre-training data … on REACT 2024” is empirically true but not evidence of a fully independent alternative to real DIAD collection. Not forced by equation identity—only a mild related-distribution loop.

full rationale

This is a systems paper, not a first-principles derivation. The generation pipeline (TDG → DADG/IAR → IFBG/IFBR) is a modular engineering construction, not a claim that some quantity is predicted from independent axioms. Main comparisons (Table 1–2) use standard external metrics (FID, FVD, LSE-C/D, CSIM, LPIPS) plus a 60-person user study against talking-face and FRG baselines; those results are not forced by construction. The only circularity-adjacent step is the downstream IHG experiment (Sec. 5.5, Table 5): IFBR is pre-trained on REACT 2024, CHAT-AVD-50k is synthesised with that IFBR, then PerFRDiff/ReactDiff (same research group) are pre-trained on the synthetic set and fine-tuned/evaluated again on REACT. That is related-distribution augmentation, not Eq. X = Eq. Y by definition, and the paper explicitly hedges (“large-scale diverse augmentation over a related interaction distribution rather than as a fully out-of-domain source”). FRCorr/FRDiv come from the authors’ prior REACT work but are applied as published metrics, not redefined to favour CHAT. No self-definitional equations, no fitted parameter renamed as prediction, no uniqueness theorem imported from the authors. Score 2 for one non-load-bearing related-distribution self-loop on the secondary data claim; main generation claims are self-contained.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 4 invented entities

CHAT is a modular systems paper. Its claims rest on standard generative-model assumptions (LLM dialogue coherence, TTS quality, video diffusion expressivity), several hand-chosen pipeline constants (interaction-word probability, boundary blend width, multi-scale injection schedule, turn-length limits), and newly named modules (IAR, IFBR/SFBG/RFBG/TCR) that are architectural compositions rather than new physical entities. Downstream transfer further assumes that synthetic dialogues produced with REACT-pretrained IFBR still add useful diversity when models are fine-tuned back on REACT.

free parameters (5)
  • P_inter (interactive-word insertion probability)
    Controls how often short listener responses are inserted into silent segments (Eq. 4); chosen by design, not derived.
  • Boundary blend half-width W and Gaussian σ=W/3
    Empirically set to W=10 frames for TCR blending (Sec. 4.3); directly affects perceived temporal continuity.
  • Multi-scale injection schedule l(t)=max(1,⌈L t/T⌉) and number of scales L
    Hand-designed progressive conditioning of RFBG diffusion; not learned from first principles.
  • RFBG denoising steps (50)
    Fixed training/inference hyperparameter for the reaction diffusion model (Sec. 5.1).
  • Dialogue turn length constraint (5–10 turns)
    Hard limit chosen so LLMs stay coherent; shapes the entire synthetic distribution.
axioms (5)
  • domain assumption Modern LLMs can produce diverse, coherent multi-turn dyadic scripts and emotion/prosody descriptors from a short scenario prompt without long-context collapse within 5–10 turns.
    Invoked throughout TDG and IAR (Sec. 4.1–4.2); performance of Gemini is taken as given.
  • domain assumption Pre-trained emotional TTS plus content/emotion Transformer fusion yields speech whose emotion labels and lip-sync are accurate enough for downstream face animation.
    DADG/IAR pipeline (Sec. 4.2) freezes TTS and treats its outputs as reliable conditions.
  • domain assumption Video diffusion conditioned on multi-scale partner face/audio features can generate listener facial behaviours that are both identity-preserving and socially responsive.
    Core premise of RFBG (Eqs. 8–9, Fig. 3); supported by REACT-style training but not independently proven.
  • domain assumption Turn-taking with rare simultaneous speech is the dominant structure of natural dyadic conversation, so silent-segment modelling plus occasional backchannels is sufficient.
    Stated in Sec. 4.2 when justifying IAR’s interactive-word insertion design.
  • standard math Standard diffusion/transformer math and cross-attention conditioning behave as in prior video-generation literature.
    Used for SilentDiff, RFBG DiT, and multi-scale CA without new proofs.
invented entities (4)
  • DIADG task no independent evidence
    purpose: Formal problem of generating multiple diverse mutually responsive dyadic audio-visual dialogue clips from one text prompt.
    Defined in Sec. 3 (Eq. 1); organises the paper’s contribution relative to prior talking-face and FRG tasks.
  • Interactive Audio Refinement (IAR) block no independent evidence
    purpose: Convert non-interactive TTS outputs into emotion-aware, backchannel-augmented paired speech.
    Named module in Sec. 4.2; composition of LLM prompting, encoders, and TTS rather than a new physical object.
  • Interactive Facial Behaviour Refinement (IFBR) with SFBG/RFBG/TCR no independent evidence
    purpose: Turn independent talking-face segments into mutually responsive, temporally continuous face-video pairs.
    Core trainable contribution in Sec. 4.3; architectural invention evaluated only inside this pipeline.
  • CHAT-AVD-50k dataset no independent evidence
    purpose: 50k synthetic DIAD pairs offered as pre-training resource for interactive head generation.
    Sec. 4.1 and 5.5; utility shown only via REACT fine-tuning gains, not external independent benchmarks yet.

pith-pipeline@v1.1.0-grok45 · 23332 in / 4297 out tokens · 42564 ms · 2026-07-12T06:58:18.977488+00:00 · methodology

0 comments
read the original abstract

Large-scale dyadic interactive audio-visual dialogue (DIAD) datasets provide fundamental data resources for developing humanoid interactive virtual agents and digital humans. However, collecting such data is time-consuming, expensive, and ethically sensitive. To address this, we propose CHAT, a new dyadic interactive audio-visual dialogue generation (DIADG) framework that generates diverse, paired, and mutually responsive speech-face dialogue clips from a single textual prompt. CHAT unifies large language models and talking face models with interactive audio and facial behaviour refinement modules, enabling the generation of aligned dyadic dialogue clips with diverse contents and facial identities. Experiments show that CHAT outperforms existing related methods designed for similar tasks under both objective and subjective evaluations. Moreover, our synthesised CHAT-AVD-50k dataset serves as effective pre-training data for downstream interactive head generation, consistently improving PerFRDiff and ReactDiff on REACT 2024. CHAT offers a scalable alternative to the costly and ethically sensitive collection of real dyadic interaction data.

Figures

Figures reproduced from arXiv: 2607.02799 by Haijun Lin, Junhao Song, Linlin Shen, Lluis Guasch, Shizhe Liu, Siyang Song, Weicheng Xie, Wei Pang, Xilin He, Yingfang Yuan, Zhongyu Yang.

Figure 1
Figure 1. Figure 1: Existing methods: (a) Condition-driven face video generation [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The CHAT pipeline. (a) TDG generates textual dialogue pairs and identity descriptors from a single user prompt P; (b) DADG converts each dialogue into inter￾active, emotion-aware audio pairs via the IAR block (Sec. 4.2); (c) IFBG synthesises mutually responsive face video pairs using a talking face model and the IFBR block (Sec. 4.3). All DIAD clips are assembled from DADG and IFBG outputs. To ensure robus… view at source ↗
Figure 3
Figure 3. Figure 3: Illustration of the proposed RFBG sub-block. identity image IDi n as well as v¯ i n (k −1) and v¯ i n (k + 1) via a video diffusion model SilentDiff as: \bar {v}_n^i(k) = \text {SilentDiff}(\bar {v}_n^i(k-1), \bar {v}_n^i(k+1), \text {ID}_n^i). (7) SilentDiff inherits the same architecture from [10] but is end-to-end optimised with other modules/blocks within our framework. This also maintains rough tem￾po… view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative comparison. Talking-face methods (Hallo3, EDTalk, SadTalker) used twice yield non-responsive pairs; FRG methods (DIM, ReactDiff) take user￾provided speaker video (blue frames). CHAT generates mutually responsive audio￾visual dialogue pairs. where 1 ≤ τ ≤ W and Fk−1 is the total number of frames in segment k − 1, and α(τ ) = 1 2 exp(−(τ − 1)2/2σ 2 ) is a half-Gaussian blending weight, equal to 0… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

78 extracted references · 19 linked inside Pith

  1. [1]

    arXiv preprint arXiv:2303.08774 (2023)

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.: GPT-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    Pith/arXiv arXiv 2023

  2. [2]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Aneja, S., Thies, J., Dai, A., Nießner, M.: Facetalk: Audio-driven motion diffusion for neural parametric head models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 21263–21273 (2024)

    2024

  3. [3]

    In: 2024 IEEE Conference Virtual Reality and 3D User Interfaces (VR)

    Bai, Z., Chen, P., Peng, X., Liu, L., Yao, N., Chen, H.: Bring your own character: A holistic solution for automatic facial animation generation of customized charac- ters. In: 2024 IEEE Conference Virtual Reality and 3D User Interfaces (VR). pp. 429–438. IEEE (2024)

    2024

  4. [4]

    arXiv preprint arXiv:2406.04904 (2024)

    Casanova, E., Davis, K., Gölge, E., Göknar, G., Gulea, I., Hart, L., Aljafari, A., Meyer, J., Morais, R., Olayemi, S., et al.: XTTS: a massively multilingual zero-shot text-to-speech model. arXiv preprint arXiv:2406.04904 (2024)

    Pith/arXiv arXiv 2024

  5. [5]

    In: Pro- ceedings of the IEEE/CVF International Conference on Computer Vision (2025)

    Chatziagapi, A., Morency, L.P., Gong, H., Zollhoefer, M., Samaras, D., Richard, A.: AV-Flow: Transforming text to audio-visual human-like interactions. In: Pro- ceedings of the IEEE/CVF International Conference on Computer Vision (2025)

    2025

  6. [6]

    Image and Vision Computing p

    Chen, K., Li, Z., Cui, F., Ni, M., Wang, S., Che, J., Liu, F., Qi, Y., Zhang, F., Liu, J., et al.: FastTalker: Real-time audio-driven talking face generation with 3d gaussian. Image and Vision Computing p. 105573 (2025)

    2025

  7. [7]

    In: 2020 twelfth international conference on quality of multimedia experience (QoMEX)

    Chinen,M.,Lim,F.S.,Skoglund,J.,Gureev,N.,O’Gorman,F.,Hines,A.:ViSQOL v3: An open source production ready objective speech and audio metric. In: 2020 twelfth international conference on quality of multimedia experience (QoMEX). pp. 1–6. IEEE (2020)

    2020

  8. [8]

    In: ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

    Choi, J., Kim, M., Park, S.J., Ro, Y.M.: Text-driven talking face synthesis by reprogramming audio-driven models. In: ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 8065–8069. IEEE (2024)

    2024

  9. [9]

    In: Proceedings of the SIGGRAPH Asia 2025 Conference Papers

    Cui, J., Chen, B., Xu, M., Shang, H., Chen, Y., Su, Q., Dong, Z., Yao, Y., Wang, J., Zhu, S.: High-fidelity dynamic portrait animation via direct preference optimiza- tion and temporal motion modulation. In: Proceedings of the SIGGRAPH Asia 2025 Conference Papers. pp. 1–10 (2025)

    2025

  10. [10]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Cui, J., Li, H., Zhan, Y., Shang, H., Cheng, K., Ma, Y., Mu, S., Zhou, H., Wang, J., Zhu, S.: Hallo3: Highly dynamic and realistic portrait image animation with video diffusion transformer. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 21086–21095 (2025)

    2025

  11. [11]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Deng, J., Guo, J., Xue, N., Zafeiriou, S.: Arcface: Additive angular margin loss for deep face recognition. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 4690–4699 (2019)

    2019

  12. [12]

    In: ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

    Deng, K., Zheng, D., Xie, J., Wang, J., Xie, W., Shen, L., Song, S.: DEGSTalk: Decomposed per-embedding gaussian fields for hair-preserving talking face synthe- sis. In: ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 1–5. IEEE (2025)

    2025

  13. [13]

    Nature Eye39(2), 215–216 (2025)

    Dihan, Q.A., Nihalani, B.R., Tooley, A.A., Elhusseiny, A.M.: Eyes on google’s notebooklm: using generative ai to create ophthalmology podcasts with a single click. Nature Eye39(2), 215–216 (2025)

    2025

  14. [14]

    arXiv preprint arXiv:2407.21783 (2024) Conversational Human Audio-visual Talking Dialogue Generation 17

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al.: The LLaMA 3 herd of models. arXiv preprint arXiv:2407.21783 (2024) Conversational Human Audio-visual Talking Dialogue Generation 17

    Pith/arXiv arXiv 2024

  15. [15]

    In: Proceed- ings of the 63rd Annual Meeting of the Association for Computational Linguistics

    Fang, Q., Zhou, Y., Guo, S., Zhang, S., Feng, Y.: LLaMA-Omni2: Llm-based real- time spoken chatbot with autoregressive streaming speech synthesis. In: Proceed- ings of the 63rd Annual Meeting of the Association for Computational Linguistics. pp. 18617–18629. ACL (2025)

    2025

  16. [16]

    arXiv preprint arXiv:2507.06261 (2025)

    Gemini Team: Gemini 2.5: Pushing the frontier with advanced reasoning, multi- modality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261 (2025)

    Pith/arXiv arXiv 2025

  17. [17]

    In: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing

    Ghosh, S., Kumar, S., Seth, A., et al.: Gama: A large audio-language model with advanced audio understanding and complex reasoning abilities. In: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. pp. 6288–6313 (2024)

    2024

  18. [18]

    Nature645(8081), 633–638 (2025)

    Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., Xu, R., Zhang, R., Ma, S., Bi, X., et al.: Deepseek-r1 incentivizes reasoning in llms through reinforcement learning. Nature645(8081), 633–638 (2025)

    2025

  19. [19]

    In: European Conference on Com- puter Vision

    Guo, Y., Yang, C., Rao, A., Agrawala, M., Lin, D., Dai, B.: Sparsectrl: Adding sparse controls to text-to-video diffusion models. In: European Conference on Com- puter Vision. p. 330–348. Berlin, Heidelberg (2024)

    2024

  20. [20]

    International Conference on Learning Representations (2024)

    Guo, Y., Yang, C., Rao, A., Liang, Z., Wang, Y., Qiao, Y., Agrawala, M., Lin, D., Dai, B.: Animatediff: Animate your personalized text-to-image diffusion mod- els without specific tuning. International Conference on Learning Representations (2024)

    2024

  21. [21]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    He, X., Luo, C., Xian, X., Li, B., Khan, M.H., Ge, Z., Xie, W., Song, S., Shen, L., Ghanem, B., et al.: SynFER: Towards boosting facial expression recognition with synthetic data. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 10184–10195 (2025)

    2025

  22. [22]

    arXiv preprint arXiv:2208.01626 (2022)

    Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y., Cohen-Or, D.: Prompt-to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.01626 (2022)

    Pith/arXiv arXiv 2022

  23. [23]

    Ad- vances in Neural Information Processing Systems30(2017)

    Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: GANs trained by a two time-scale update rule converge to a local nash equilibrium. Ad- vances in Neural Information Processing Systems30(2017)

    2017

  24. [24]

    In: Proceedings of the 33rd ACM Interna- tional Conference on Multimedia

    Huang, J., Yu, Z.: Multiple appropriate facial reaction generation based on multi- view transformation of speaker video. In: Proceedings of the 33rd ACM Interna- tional Conference on Multimedia. pp. 13992–13996 (2025)

    2025

  25. [25]

    arXiv preprint arXiv:2506.17298 (2025)

    Inception Lab: Mercury: Ultra-fast language models based on diffusion. arXiv preprint arXiv:2506.17298 (2025)

    Pith/arXiv arXiv 2025

  26. [26]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Jang, Y., Kim, J.H., Ahn, J., Kwak, D., Yang, H.S., Ju, Y.C., Kim, I.H., Kim, B.Y., Chung, J.S.: Faces that speak: Jointly synthesising talking face and speech from text. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 8818–8828 (2024)

    2024

  27. [27]

    arXiv preprint arXiv:2512.20296 (2025)

    Kim, J.H., Ahn, J., Kwak, D., Chung, J.S., Watanabe, S.: TAVID: Text-driven audio-visual interactive dialogue generation. arXiv preprint arXiv:2512.20296 (2025)

    arXiv 2025

  28. [28]

    In: Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition

    Kim, M., Liu, F., Jain, A., Liu, X.: DCFace: Synthetic face generation with dual condition diffusion model. In: Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition. pp. 12715–12725 (2023)

    2023

  29. [29]

    Song et al

    Kondratyuk, D., Yu, L., Gu, X., Lezama, J., Huang, J., Schindler, G., Hornung, R., Birodkar, V., Yan, J., Chiu, M.C., Somandepalli, K., Akbari, H., Alon, Y., Cheng, Y., Dillon, J.V., Gupta, A., Hahn, M., Hauth, A., Hendon, D., Martinez, A., Minnen, D., Sirotenko, M., Sohn, K., Yang, X., Adam, H., Yang, M.H., Essa, I., 18 J. Song et al. Wang, H., Ross, D.A...

    2024

  30. [30]

    arXiv preprint arXiv:2505.19978 (2025)

    Koudounas, A., La Quatra, M., Baralis, E.: DeepDialogue: A multi-turn emotionally-rich spoken dialogue dataset. arXiv preprint arXiv:2505.19978 (2025)

    Pith/arXiv arXiv 2025

  31. [31]

    Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies8(2), 1–36 (2024)

    Kovačević, N., Holz, C., Gross, M., Wampfler, R.: The personality dimensions GPT-3 expresses during human-chatbot interactions. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies8(2), 1–36 (2024)

    2024

  32. [32]

    In: Proceedings of IEEE pacific rim conference on communications comput- ers and signal processing

    Kubichek, R.: Mel-cepstral distance measure for objective speech quality assess- ment. In: Proceedings of IEEE pacific rim conference on communications comput- ers and signal processing. vol. 1, pp. 125–128. IEEE (1993)

    1993

  33. [33]

    arXiv preprint arXiv:2506.23552 (2025)

    Kwon, M., Shin, J., Jung, J., Park, J., Uh, Y.: JAM-Flow: Joint audio-motion synthesis with flow matching. arXiv preprint arXiv:2506.23552 (2025)

    Pith/arXiv arXiv 2025

  34. [34]

    arXiv preprint arXiv:2412.09262 (2024)

    Li, C., Zhang, C., Xu, W., Lin, J., Xie, J., Feng, W., Peng, B., Chen, C., Xing, W.: LatentSync: Taming audio-conditioned latent diffusion models for lip sync with syncnet supervision. arXiv preprint arXiv:2412.09262 (2024)

    Pith/arXiv arXiv 2024

  35. [35]

    In: Advances in Neural Information Processing Systems

    Li,J.N.,Tu,Q.,Mao,C.,Yu,Z.,Wen,J.R.,Yan,R.:StreamingDialogue:Prolonged dialogue learning via long context compression with minimal losses. In: Advances in Neural Information Processing Systems. vol. 37, pp. 86074–86101 (2024)

    2024

  36. [36]

    Artificial Intelligence308, 103714 (2022)

    Liang, Y., Meng, F., Zhang, Y., Chen, Y., Xu, J., Zhou, J.: Emotional conversation generation with heterogeneous graph neural network. Artificial Intelligence308, 103714 (2022)

    2022

  37. [37]

    In: 2024 IEEE 18th International Conference on Automatic Face and Gesture Recog- nition (FG)

    Liu, Z., Liang, C., Wang, J., Zhang, H., Liu, Y., Zhang, C., Gui, J., Wang, S.: One- to-many appropriate reaction mapping modeling with discrete latent variable. In: 2024 IEEE 18th International Conference on Automatic Face and Gesture Recog- nition (FG). pp. 1–5. IEEE (2024)

    2024

  38. [38]

    arXiv preprint arXiv:2502.04328 (2025)

    Liu, Z., Dong, Y., Wang, J., Liu, Z., Hu, W., Lu, J., Rao, Y.: Ola: Pushing the fron- tiers of omni-modal language model with progressive modality alignment. arXiv preprint arXiv:2502.04328 (2025)

    Pith/arXiv arXiv 2025

  39. [39]

    IEEE Transactions on Visualization and Computer Graphics (2024)

    Luo, C., Song, S., Xie, W., Spitale, M., Ge, Z., Shen, L., Gunes, H.: ReactFace: Online multiple appropriate facial reaction generation in dyadic interactions. IEEE Transactions on Visualization and Computer Graphics (2024)

    2024

  40. [40]

    In: Proceedings of the 33rd ACM Interna- tional Conference on Multimedia

    Luo, C., Song, S., Yan, S., Yu, Z., Ge, Z.: ReactDiff: Fundamental multiple appro- priate facial reaction diffusion model. In: Proceedings of the 33rd ACM Interna- tional Conference on Multimedia. pp. 5607–5616 (2025)

    2025

  41. [41]

    arXiv preprint arXiv:2406.01900 (2024)

    Ma, Y., Liu, H., Wang, H., Pan, H., He, Y., Yuan, J., Zeng, A., Cai, C., Shum, H.Y., Liu, W., et al.: Follow-Your-Emoji: Fine-controllable and expressive freestyle portrait animation. arXiv preprint arXiv:2406.01900 (2024)

    Pith/arXiv arXiv 2024

  42. [42]

    In: Proceedings of the 33rd ACM International Conference on Multimedia

    Mao, Q., Wu, Q., Liu, N., Ding, Y., Gao, L.: Scattering-conditioned diffusion mod- els for multiple appropriate facial reaction generation. In: Proceedings of the 33rd ACM International Conference on Multimedia. pp. 13985–13991 (2025)

    2025

  43. [43]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Ng, E., Romero, J., Bagautdinov, T., Bai, S., Darrell, T., Kanazawa, A., Richard, A.: From audio to photoreal embodiment: Synthesizing humans in conversations. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1001–1010 (2024)

    2024

  44. [44]

    In: 2024 IEEE 18th International Conference on Automatic Face and Gesture Recognition (FG)

    Nguyen, M.D., Yang, H.J., Ho, N.H., Kim, S.H., Kim, S., Shin, J.E.: Vector quan- tized diffusion models for multiple appropriate reactions generation. In: 2024 IEEE 18th International Conference on Automatic Face and Gesture Recognition (FG). pp. 1–5. IEEE (2024) Conversational Human Audio-visual Talking Dialogue Generation 19

    2024

  45. [45]

    In: Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition

    Niklaus, S., Liu, F.: Softmax splatting for video frame interpolation. In: Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 5437–5446 (2020)

    2020

  46. [46]

    In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Pang, Y., Zhang, Y., Quan, W., Fan, Y., Cun, X., Shan, Y., Yan, D.m.: DPE: Dis- entanglement of pose and expression for general video portrait editing. In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 427–436 (2023)

    2023

  47. [47]

    In: European Con- ference on Computer Vision

    Papantoniou, F.P., Lattas, A., Moschoglou, S., Deng, J., Kainz, B., Zafeiriou, S.: Arc2Face: A foundation model for id-consistent human faces. In: European Con- ference on Computer Vision. pp. 241–261. Springer (2024)

    2024

  48. [48]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Peng, Z., Fan, Y., Wu, H., Wang, X., Liu, H., He, J., Fan, Z.: DualTalk: Dual- speaker interaction for 3d talking head conversations. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 21055– 21064 (2025)

    2025

  49. [49]

    In: Proceedings of the 28th ACM international conference on multimedia

    Prajwal, K., Mukhopadhyay, R., Namboodiri, V.P., Jawahar, C.: A lip sync expert is all you need for speech to lip generation in the wild. In: Proceedings of the 28th ACM international conference on multimedia. pp. 484–492 (2020)

    2020

  50. [50]

    arXiv preprint arXiv:2503.21144 (2025)

    Qi, J., Ji, C., Xu, S., Zhang, P., Zhang, B., Bo, L.: ChatAnyone: Stylized real-time portrait video generation with hierarchical motion diffusion model. arXiv preprint arXiv:2503.21144 (2025)

    Pith/arXiv arXiv 2025

  51. [51]

    arXiv preprint arXiv:2505.09388 (2025)

    Qwen Team: Qwen3 technical report. arXiv preprint arXiv:2505.09388 (2025)

    Pith/arXiv arXiv 2025

  52. [52]

    arXiv preprint arXiv:2106.04624 (2021)

    Ravanelli, M., Parcollet, T., Plantinga, P., Rouhe, A., Cornell, S., Lugosch, L., Subakan, C., Dawalatabad, N., Heba, A., Zhong, J., et al.: SpeechBrain: A general- purpose speech toolkit. arXiv preprint arXiv:2106.04624 (2021)

    Pith/arXiv arXiv 2021

  53. [53]

    In: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics

    Sharma, D.V., Ekbote, V., Gupta, A.: IndicSynth: A large-scale multilingual syn- thetic speech dataset for low-resource indian languages. In: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics. pp. 22037–22060 (2025)

    2025

  54. [54]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision (Oc- tober 2025)

    Siniukov, M., Chang, D., Tran, M., Gong, H., Chaubey, A., Soleymani, M.: Di- TaiListener: Controllable high fidelity listener video generation with diffusion. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (Oc- tober 2025)

    2025

  55. [55]

    IEEE Transactions on Affective Computing14(4), 3048–3065 (2022)

    Song, S., Shao, Z., Jaiswal, S., Shen, L., Valstar, M., Gunes, H.: Learning person- specific cognition from facial reactions for automatic personality recognition. IEEE Transactions on Affective Computing14(4), 3048–3065 (2022)

    2022

  56. [56]

    ACM Multimedia 2025 (2025)

    Song, S., Spitale, M., Kong, X., Zhu, H., Luo, C., Palmero, C., Barquero, G., Escalera, S., Valstar, M., Daoudi, M., et al.: REACT 2025: the third multiple appropriate facial reaction generation challenge. ACM Multimedia 2025 (2025)

    2025

  57. [57]

    In: Proceedings of the 31st ACM International Conference on Multimedia

    Song, S., Spitale, M., Luo, C., Barquero, G., Palmero, C., Escalera, S., Valstar, M., Baur, T., Ringeval, F., André, E., et al.: REACT2023: The first multiple ap- propriate facial reaction generation challenge. In: Proceedings of the 31st ACM International Conference on Multimedia. pp. 9620–9624 (2023)

    2023

  58. [58]

    In: 2024 IEEE 18th International Conference on Automatic Face and Gesture Recognition (FG)

    Song, S., Spitale, M., Luo, C., Palmero, C., Barquero, G., Zhu, H., Escalera, S., Val- star, M., Baur, T., Ringeval, F., et al.: React 2024: the second multiple appropriate facial reaction generation challenge. In: 2024 IEEE 18th International Conference on Automatic Face and Gesture Recognition (FG). pp. 1–5. IEEE (2024)

    2024

  59. [59]

    IEEE Transactions on Visualization and Computer Graphics (2024) 20 J

    Song, W., Wang, X., Zheng, S., Li, S., Hao, A., Hou, X.: TalkingStyle: personalized speech-driven 3d facial animation with style preservation. IEEE Transactions on Visualization and Computer Graphics (2024) 20 J. Song et al

    2024

  60. [60]

    In: European Conference on Computer Vision

    Tan, S., Ji, B., Bi, M., Pan, Y.: EDTalk: Efficient disentanglement for emotional talking head synthesis. In: European Conference on Computer Vision. pp. 398–416. Springer (2024)

    2024

  61. [61]

    In: European Conference on Computer Vision

    Tran, M., Chang, D., Siniukov, M., Soleymani, M.: DIM: Dyadic interaction mod- eling for social behavior generation. In: European Conference on Computer Vision. pp. 484–503. Springer (2024)

    2024

  62. [62]

    arXiv preprint arXiv:1812.01717 (2018)

    Unterthiner, T., Van Steenkiste, S., Kurach, K., Marinier, R., Michalski, M., Gelly, S.: Towards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01717 (2018)

    Pith/arXiv arXiv 2018

  63. [63]

    arXiv preprint arXiv:2303.02490 (2023)

    Wang, B., Vastola, J.J.: Diffusion models generate images like painters: an analyt- ical theory of outline first, details later. arXiv preprint arXiv:2303.02490 (2023)

    Pith/arXiv arXiv 2023

  64. [64]

    In: Proceedings of the 33rd ACM International Conference on Multimedia

    Wang, P., Xue, P., Liu, X., Ji, T.: Explaining listener reactions: Personality-guided facial response generation with cross-modal attention. In: Proceedings of the 33rd ACM International Conference on Multimedia. pp. 13997–14003 (2025)

    2025

  65. [65]

    arXiv preprint arXiv:2410.03017 (2024)

    Wang, R.E., Ribeiro, A.T., Robinson, C.D., Loeb, S., Demszky, D.: Tutor CoPilot: A human-ai approach for scaling real-time expertise. arXiv preprint arXiv:2410.03017 (2024)

    Pith/arXiv arXiv 2024

  66. [66]

    In: Advances in Neural Information Processing Systems

    Wang, Z., Zhang, P., Qi, J., Xu, S., Zhang, B., et al.: OmniTalker: One-shot real- time text-driven talking audio-video generation with multimodal style mimicking. In: Advances in Neural Information Processing Systems. vol. 38, pp. 21834–21862 (2025)

    2025

  67. [67]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Xiang, J., Guo, Y., Hu, L., Guo, B., Yuan, Y., Zhang, J.: Expressive talking hu- man from single-image with imperfect priors. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 10398–10409 (2025)

    2025

  68. [68]

    arXiv preprint arXiv:2404.10667 (2024)

    Xu, S., Chen, G., Guo, Y.X., Yang, J., Li, C., Zang, Z., Zhang, Y., Tong, X., Guo, B.: VASA-1: Lifelike audio-driven talking faces generated in real time. arXiv preprint arXiv:2404.10667 (2024)

    Pith/arXiv arXiv 2024

  69. [69]

    IEEE Transactions on Affective Computing (2026)

    Xu, T., Spitale, M., Tang, H., Liu, L., Gunes, H., Song, S.: Reversible graph neural network-based reaction distribution learning for multiple appropriate facial reac- tions generation. IEEE Transactions on Affective Computing (2026)

    2026

  70. [70]

    Visual Intelligence2(1), 24 (2024)

    Yan, Y., Zhou, Z., Wang, Z., Gao, J., Yang, X.: DialogueNerf: Towards realis- tic avatar face-to-face conversation video generation. Visual Intelligence2(1), 24 (2024)

    2024

  71. [71]

    arXiv preprint arXiv:2406.12793 (2024)

    Zeng, A., Xu, B., Wang, B., Zhang, C., Yin, D., Zhang, D., Rojas, D., Feng, G., Zhao, H., et al.: ChatGLM: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793 (2024)

    Pith/arXiv arXiv 2024

  72. [72]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 586–595 (2018)

    2018

  73. [73]

    In: International Conference on Learning Representations (2025)

    Zhang, S., Fang, Q., Yang, Z., Feng, Y.: LLaVA-Mini: Efficient image and video large multimodal models with one vision token. In: International Conference on Learning Representations (2025)

    2025

  74. [74]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Zhang, W., Cun, X., Wang, X., Zhang, Y., Shen, X., Guo, Y., Shan, Y., Wang, F.: SadTalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 8652–8661 (2023)

    2023

  75. [75]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Zhang, Z., Li, L., Ding, Y., Fan, C.: Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3661–3670 (2021) Conversational Human Audio-visual Talking Dialogue Generation 21

    2021

  76. [76]

    In: Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion

    Zhao, S., Hong, F.T., Huang, X., Xu, D.: Synergizing motion and appearance: Multi-scale compensatory codebooks for talking head video generation. In: Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion. pp. 26232–26241 (2025)

    2025

  77. [77]

    In: Proceedings of the 32nd ACM International Conference on Multimedia (2024)

    Zhu, H., Kong, X., Xie, W., Huang, X., Shen, L., Liu, L., Gunes, H., Song, S.: Per- FRDiff: Personalised weight editing for multiple appropriate facial reaction gener- ation. In: Proceedings of the 32nd ACM International Conference on Multimedia (2024)

    2024

  78. [78]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Zhu, Y., Zhang, L., Rong, Z., Hu, T., Liang, S., Ge, Z.: INFP: Audio-driven inter- active head generation in dyadic conversations. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10667–10677 (2025)

    2025