Pith. sign in

REVIEW 3 major objections 7 minor 1 cited by

Predicting Turn-Taking and Backchannel in Human-Machine Conversations Using Linguistic, Acoustic, and Visual Signals

T0 review · 3 major / 7 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that fusing text, audio, and face video predicts turn-taking and backchannel moments better than earlier baselines on a new 210-hour face-to-face conversation dataset.

desk verdict A genuinely useful tri-modal dataset and a flexible fusion baseline, but the headline numbers rest on automatic proxy labels that the manual validation never actually checks. read the letter →

arxiv 2505.12654 v2 pith:G62RJPVT submitted 2025-05-19 cs.CL cs.AI

Yuxin Lin , Yinglin Zheng , Ming Zeng , Wangzheng Shi This is my paper · ORCID
classification cs.CLcs.AI
keywords turn-takingpredictionbackchannelmultimodalfusionhuman-machineconversationface-to-facedatasetfull-duplexdialoguelow-rankactivespeakerdetection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that a conversational system can predict, word by word, whether the current speaker will keep talking, a listener should take the floor, or a short acknowledgment like "I see" is due. To make that test possible, it builds a 210-hour face-to-face conversation dataset with word-level labels and a model that fuses text, audio, and cropped face video. On this dataset the fused model beats earlier text-only, text-audio, and tri-modal baselines, raising F1 by about 10 percent for turn-taking and 33 percent for backchannel prediction. If the result holds, spoken interfaces can time their responses like human listeners instead of waiting for an explicit end-of-speech signal.

What carries the argument

The load-bearing component is a flexible low-rank multimodal fusion module. It starts from the tensor outer product of the three uni-modal features and decomposes that weight tensor into a sum of rank-one factors, one per modality, so the fused representation is an element-wise product of per-modality projected features. An indicator function substitutes a constant one for any missing modality, reducing the same fused expression to a bi-modal or uni-modal form, and random modality dropout during training makes the tri-modal model robust to missing inputs at inference. This module is what lets the framework claim that any combination of text, audio, and video works after a single training run.

What would settle it

Re-annotate a random sample of MM-F2F clips with trained human raters who mark the moment a turn becomes complete and every suitable backchannel, then evaluate the same model on those human labels; if agreement between the pipeline labels and human labels is low, or if the model's F1 on the human-labeled subset drops far below the reported 0.811 and 0.906, the claimed improvements would be specific to the proxy task rather than to natural conversational timing.

Watch

Extended reading notes

Core claim

The paper's central claim is that linguistic, acoustic, and visual face signals are jointly sufficient to predict keep, turn-taking, and backchannel labels at word level in dyadic face-to-face conversation, and that a low-rank fusion that preserves inter-modality products while tolerating missing modalities outperforms earlier fusion designs. The authors support this with a fully automatic collection pipeline: speech recognition supplies word-level transcripts and timestamps, speaker verification and active speaker detection assign each utterance to one of the two faces, the last word of each detected utterance is labeled TURN, and a fixed vocabulary marks BACKCHANNEL. The resulting MM-F2F dataset spans about 1.5 million words, 20 million frames, 169,029 utterances, and 955 speakers. With GPT-2, HuBERT, and VideoMAE as uni-modal encoders, the tri-modal model reaches 0.823 accuracy and F1-scores of 0.806 for keep, 0.811 for turn-taking, and 0.906 for backchannel prediction, exceeding reimplemented baselines by roughly 0.1 on keep and turn and more than 0.2 on backchannel.

Load-bearing premise

The load-bearing assumption is that the automatic annotations match real conversational moments: TURN is the last word of each speech-recognition sentence and BACKCHANNEL is any word in a fixed list, so the reported gains are for predicting those proxies.

Editorial extensions

If this is right

  • If the reported results hold, spoken dialogue systems can decide turn-taking and backchannel timing from the speaker's words, voice, and face rather than from an explicit end-of-turn signal or a voice-activity threshold.
  • The same trained model can be deployed with only text and audio, only audio and video, or any other pair, making the approach usable in phone calls, video calls, and voice-only devices without retraining.
  • Visual cues from the face carry information that text and audio alone miss, especially for backchannels, so full-duplex systems with a camera should include the face channel.
  • Privacy-preserving de-identification of faces, voices, and backgrounds does not substantially degrade prediction, which supports building larger socially useful conversational datasets from in-the-wild videos.
  • The dataset itself provides a benchmark for word-level keep, turn, and backchannel prediction in dyadic face-to-face conversation, which earlier resources lacked.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because TURN and BACKCHANNEL are defined by ASR sentence boundaries and a fixed word list, the 10 percent and 33 percent gains are proven for those proxies; whether they transfer to what a human listener would call the right moment to speak or acknowledge is an open question the paper does not settle.
  • The backchannel vocabulary labels may overlap heavily with short common tokens, so part of the backchannel gain could come from lexical regularity rather than from genuinely multimodal timing; a test that removes the vocabulary prior would separate these.
  • The flexible fusion idea is not tied to the three encoders tested here; the same indicator-function low-rank product could carry additional signals such as body motion, gaze, or gesture.
  • A natural stress test is to deploy the audio-plus-video version of the model in a live spoken-dialogue system with no text input and measure whether users perceive the timing as appropriate rather than merely whether the proxy label is matched.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The manuscript introduces MM-F2F, a 210-hour, 1.5M-word English dyadic video dataset with word-level KEEP/TURN/BACKCHANNEL labels produced by an automatic pipeline (WhisperX ASR, speaker clustering, TalkNet active speaker detection, and rule-based annotation: TURN is the last word of an ASR sentence, BACKCHANNEL is a match against a fixed vocabulary). It then proposes a tri-modal turn-taking/backchannel prediction framework with a low-rank flexible fusion module and random modality dropout, supporting any combination of text, audio, and video inputs. On the authors' own test split, the tri-modal model reports F1 = 0.811 for TURN and 0.906 for BACKCHANNEL, exceeding the reimplemented baselines and yielding the abstract's claimed 10% and 33% relative F1 improvements.

Significance. The dataset is potentially a large-scale resource: it is an order of magnitude larger than EgoCom and is the first dyadic face-to-face corpus with text, audio, video, and backchannel annotations; the authors release code and dataset and include a useful de-identification study (Appendix A.1). The fusion design is clean, and the random modality dropout result (Table 6) demonstrates a practical robustness benefit for missing modalities. However, the significance of the state-of-the-art claim depends entirely on the validity of the automatically derived TURN/BACKCHANNEL labels; the current manual validation does not verify those labels, and all comparisons are point estimates on an internal benchmark. The work is therefore a solid engineering contribution whose headline empirical claim is not yet established.

major comments (3)
  1. [§3.1.5 and §A.3] The annotation rule in §3.1.5 defines TURN as the last word of each WhisperX sentence-level ASR segment and BACKCHANNEL by matching an unlisted fixed vocabulary. Because WhisperX segments on silence and punctuation rather than conversational turn structure, a mid-utterance pause creates a spurious TURN and a fluent speaker change without a pause is missed. The manual double-check in §A.3 asks validators only about frame composition, active-speaker boxes, subtitle accuracy, and naturalness (Q1-Q4); no question checks whether the TURN or BACKCHANNEL label itself is correct. Consequently the §3.1.5 claim of better than 95% precision is unsupported for the labels actually used in training and evaluation, and the reported gains (e.g., Turn F1 0.811 vs 0.739 and BC F1 0.906 vs 0.680 in Table 5) may largely measure prediction of ASR segment boundaries plus a word list rather than the conversational phenomena named in the title. I request a sample-based human agreement study on the TURN/BACKCHANNEL labels themselves (e.g., several hundred words annotated by independent raters against the video) and publication of the backchannel vocabulary.
  2. [§5.4 and Tables 3-6] All reported results are single point estimates on the authors' own test split, with no standard deviations across random seeds, no confidence intervals, and no significance tests. The baselines in Table 5 are reimplemented by the authors on the same automatic labels, so the state-of-the-art claim is relative to an internal, self-constructed benchmark; without variance estimates or an evaluation on an independently annotated corpus (e.g., a hand-labeled subset of MM-F2F or an existing public corpus such as EgoCom or FTAD), the 10%/33% improvement of the abstract is not distinguishable from training noise or from artifacts of the reimplementation. Please report mean +/- standard deviation over at least three seeds and, if feasible, validate on externally labeled data.
  3. [§7 and Fig. 5] The limitation section acknowledges a failure case in which a speaker pauses with semantically incomplete content and the model predicts turn-taking instead of backchannel. Under the §3.1.5 definition, however, a pause is exactly what creates a TURN label, so this failure case is not merely a model deficiency but a direct consequence of the label proxy. The section should state this explicitly, or provide evidence that WhisperX sentence boundaries align with genuine turn transitions in these cases.
minor comments (7)
  1. [Eq. (4)] The symbols I_T, I_A, I_V are described as indicator functions but are used as identity-or-constant-one mappings; please rename them to avoid confusion with set indicators.
  2. [§3.1.1 vs §A.2] Section 3.1.1 says conversations containing personal information were manually removed, while Appendix A.2 says text dialogue content is de-identified; please clarify whether the released transcripts are original or rewritten.
  3. [Fig. 5] The caption 'GT. Pred.' suggests a comparison, but no predicted labels or probability values are shown in the figure; please clarify what is being visualized.
  4. [§4.3.2] For single-modality input the paper says the model defaults to the uni-modal encoder head, but Eq. (4) with one active modality would also produce a fused representation; please explain why the uni-modal head is preferred.
  5. [Table 6] The w/o RMDT bi-modal rows collapse to near-zero F1 for some classes (e.g., T+V Turn F1 = 0.001 and BC F1 = 0.005), which suggests degenerate prediction; please report class distributions or confusion matrices for these rows.
  6. [Abstract] The abstract's '10% increase in F1-score on turn-taking and a 33% increase on backchannel prediction' are relative improvements over the Wang et al. baseline from Table 5; please state this explicitly.
  7. [Related Work] The related-work section mentions 'Chang et al.' and 'Yang et al.' without reference numbers; please add the corresponding citations.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: model predictions are not used to create labels, baselines are independently reimplemented on the same benchmark, and no self-citation chain is load-bearing.

full rationale

The paper's central empirical claim is that its tri-modal fusion model outperforms prior models on its own MM-F2F benchmark. This is not circular: the automatic annotation pipeline (WhisperX segmentation, fixed backchannel vocabulary, speaker clustering, TalkNet) is applied before model training and does not use model predictions; train and test labels come from the same external pipeline for all compared methods. The SOTA claim is relative to a new dataset, but the baselines (TurnGPT, Wang et al., Kurata et al.) are reimplemented from published papers and evaluated under identical conditions. The fusion module is adapted from published low-rank fusion (Liu et al. 2018) and compared with concatenation and GMF, so no result is obtained by construction. There are no self-citations by these authors used as evidence. The proxy-label concern—TURN is defined as the last word of each WhisperX sentence segment and BACKCHANNEL by a fixed vocabulary, with manual validation in Sec. A.3 checking framing, speaker detection, subtitles, and naturalness but not the TURN/BC labels themselves—is a question of construct validity and label quality, not circularity, because no model output is fed back into the labels. Therefore no circular step satisfying the quoted-reduction standard is present.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new theoretical entities. The load-bearing assumptions are the proxy labels (ASR sentence boundaries and a fixed backchannel vocabulary), the dataset selection constraints, and the reliability of speaker/active-speaker tools. The main free parameters are the fusion rank, frame window, and data-selection thresholds.

free parameters (4)
  • low-rank fusion rank r = 16
    Set in Section 5.1.2; chosen by hand, controls capacity of the fusion module.
  • video frame window n = 16
    The visual encoder processes the last 16 frames of each word clip (Section 4.2.3); chosen without reported sensitivity analysis.
  • face detection confidence threshold = 0.95
    Used to select videos with exactly two frontal faces (Appendix A.2); affects dataset composition.
  • voiceprint perturbation magnitude = 20% of standard deviation
    Applied per-dimension to voice ID latent code during de-identification (Section 3.1.1); not varied.
assumptions (5)
  • domain assumption WhisperX sentence boundaries correspond to natural turn-taking points
    TURN labels are the last word of each ASR utterance (Section 3.1.5). The paper provides no validation of this proxy against human-annotated turn-taking.
  • domain assumption Backchannel vocabulary from prior work covers backchannel acts in this dataset
    BACKCHANNEL is annotated by matching words to a vocabulary following Ekstedt and Skantze (2020) (Section 3.1.5).
  • domain assumption Two frontal faces in a split-screen video are a valid proxy for face-to-face human-machine conversation
    Dataset selection criteria (Appendix A.2) restrict to left-right, camera-facing dyads.
  • domain assumption De-identification preserves the conversational cues needed for the task
    Tested in Appendix A.1, but only on the automatic metrics; human judgment of naturalness is used for filtering, not for label correctness.
  • domain assumption Active speaker detection and speaker clustering correctly assign utterances to speakers
    TalkNet and KMeans are used without reported accuracy on this corpus (Section 3.1.3-3.1.4).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Predicting Turn-Taking and Backchannel in Human-Machine Conversations Using Linguistic, Acoustic, and Visual Signals." pith.science (2026). https://pith.science/paper/G62RJPVT

@misc{pith2026250512654,
  author       = {Pith},
  title        = {Pith review of: Predicting Turn-Taking and Backchannel in Human-Machine Conversations Using Linguistic, Acoustic, and Visual Signals},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G62RJPVT}},
  note         = {Machine review of arXiv:2505.12654}
}
read the original abstract

This paper addresses the gap in predicting turn-taking and backchannel actions in human-machine conversations using multi-modal signals (linguistic, acoustic, and visual). To overcome the limitation of existing datasets, we propose an automatic data collection pipeline that allows us to collect and annotate over 210 hours of human conversation videos. From this, we construct a Multi-Modal Face-to-Face (MM-F2F) human conversation dataset, including over 1.5M words and corresponding turn-taking and backchannel annotations from approximately 20M frames. Additionally, we present an end-to-end framework that predicts the probability of turn-taking and backchannel actions from multi-modal signals. The proposed model emphasizes the interrelation between modalities and supports any combination of text, audio, and video inputs, making it adaptable to a variety of realistic scenarios. Our experiments show that our approach achieves state-of-the-art performance on turn-taking and backchannel prediction tasks, achieving a 10% increase in F1-score on turn-taking and a 33% increase on backchannel prediction. Our dataset and code are publicly available online to ease of subsequent research.

Figures

Figures reproduced from arXiv: 2505.12654 by the authors.

Figure 1
Figure 1. In face-to-face conversation scenarios, the computer determines keeping, turn-taking and backchannel actions according to the user’s linguistic, acoustic and visual signals (as shown above the line). the appropriate response time. Besides turn-taking, human conversations often consist of short overlap￾ping statements. For example, the listener might say “I see” to express understanding or respond with an “hmm” to in… view at source ↗
Figure 2
Figure 2. Samples of EgoCom vs. our MM-F2F dataset [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Architecture of our proposed multi-modal [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Qualitative results on visual modal ablation. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Failure case. When the speaker pauses to think [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Our automatic data annotation pipeline. Video Transcription. We employed WhisperX (Bain et al., 2023), built upon the whisper-large￾v3 model, to generate automatic transcriptions of the videos. The output included word-level and sentence-level transcriptions, along wit…
Figure 7
Figure 7. Figure 7: We developed a simple data validation system [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Multimodal Voice Activity Projection for Turn-Taking in Social Robots with Voice-Activity-Related Pretrained Encoders

    cs.RO 2026-07 conditional novelty 5.0 of 10

    Pretrained audio-visual speech encoders adapted with LoRA improve multimodal voice activity projection for turn-taking prediction across multiple languages and a robot mediation corpus.

Reference graph

Works this paper leans on

46 extracted references · 19 canonical work pages · cited by 1 Pith paper

  1. [1]

    Anne H Anderson, Miles Bader, Ellen Gurman Bard, Elizabeth Boyle, Gwyneth Doherty, Simon Garrod, Stephen Isard, Jacqueline Kowtko, Jan McAllister, Jim Miller, et al. 1991. The hcrc map task corpus. Language and speech, 34(4):351--366

    1991

  2. [2]

    Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33:12449--12460

    2020

  3. [3]

    Max Bain, Jaesung Huh, Tengda Han, and Andrew Zisserman. 2023. Whisperx: Time-accurate speech transcription of long-form audio. INTERSPEECH 2023

    2023

  4. [4]

    Shuo-yiin Chang, Bo Li, Tara N Sainath, Chao Zhang, Trevor Strohman, Qiao Liang, and Yanzhang He. 2022. Turn-taking prediction for natural conversational speech. arXiv preprint arXiv:2208.13321

    arXiv 2022

  5. [5]

    Kehan Chen, Zezhong Li, Suyang Dai, Wei Zhou, and Haiqing Chen. 2021. Human-to-human conversation dataset for learning fine-grained turn-taking action. In Interspeech, pages 3231--3235

    work page 2021

  6. [6]

    Jan-Peter De Ruiter, Holger Mitterer, and Nick J Enfield. 2006. Projecting the end of a speaker's turn: A cognitive cornerstone of conversation. Language, 82(3):515--535

    work page 2006

  7. [7]

    Alexandre D \'e fossez, Laurent Mazar \'e , Manu Orsini, Am \'e lie Royer, Patrick P \'e rez, Herv \'e J \'e gou, Edouard Grave, and Neil Zeghidour. 2024. Moshi: a speech-text foundation model for real-time dialogue. arXiv preprint arXiv:2410.00037

    arXiv 2024

  8. [8]

    Jiankang Deng, Jia Guo, Tongliang Liu, Mingming Gong, and Stefanos Zafeiriou. 2020. Sub-center arcface: Boosting face recognition by large-scale noisy web faces. In Proceedings of the IEEE Conference on European Conference on Computer Vision

    work page 2020

Show all 46 references
  1. [9]

    Jiankang Deng, Jia Guo, Zhou Yuxiang, Jinke Yu, Irene Kotsia, and Stefanos Zafeiriou. 2019. Retinaface: Single-stage dense face localisation in the wild. In arxiv

    2019

  2. [10]

    Jacob Devlin. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

    2018 arXiv

  3. [11]

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint...

    2020 arXiv

  4. [12]

    Erik Ekstedt and Gabriel Skantze. 2020. Turngpt: a transformer-based language model for predicting turn-taking in spoken dialog. arXiv preprint arXiv:2010.10874

    2020 arXiv

  5. [13]

    Erik Ekstedt and Gabriel Skantze. 2022 a . How much does prosody help turn-taking? investigations using voice activity projection models. arXiv preprint arXiv:2209.05161

    2022 arXiv

  6. [14]

    Erik Ekstedt and Gabriel Skantze. 2022 b . Voice activity projection: Self-supervised learning of turn-taking events. arXiv preprint arXiv:2205.09812

    2022 arXiv

  7. [15]

    John J Godfrey, Edward C Holliman, and Jane McDaniel. 1992. Switchboard: Telephone speech corpus for research and development. In Acoustics, speech, and signal processing, ieee international conference on, volume 1, pages 517--520. IEEE Computer Society

    1992

  8. [16]

    Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processin...

    2021

  9. [17]

    Keith Ito and Linda Johnson. 2017. The lj speech dataset. https://keithito.com/LJ-Speech-Dataset/

    2017

  10. [18]

    Adam Kendon. 1967. Some functions of gaze-direction in social interaction. Acta psychologica, 26:22--63

    1967

  11. [19]

    JiWoo Kim, Minsuk Chang, and JinYeong Bak. 2025. Beyond turn-taking: Introducing text-based overlap into human-llm interactions. arXiv preprint arXiv:2501.18103

    2025 arXiv

  12. [20]

    Fuma Kurata, Mao Saeki, Shinya Fujie, and Yoichi Matsuyama. 2023. Multimodal turn-taking model using visual cues for end-of-utterance prediction in spoken dialogue systems. Proc. Interspeech 2023, pages 2658--2662

    2023

  13. [21]

    Chi-Chun Lee and Shrikanth Narayanan. 2010. Predicting interruptions in dyadic spoken interactions. In 2010 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 5250--5253. IEEE

    2010

  14. [22]

    Jingyi Li, Weiping Tu, and Li Xiao. 2023. Freevc: Towards high-quality text-free one-shot voice conversion. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1--5. IEEE

    2023

  15. [23]

    Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. Dailydialog: A manually labelled multi-turn dialogue dataset. arXiv preprint arXiv:1710.03957

    2017 arXiv

  16. [24]

    Zhun Liu, Ying Shen, Varun Bharadhwaj Lakshminarasimhan, Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. 2018. Efficient low-rank multimodal fusion with modality-specific factors. arXiv preprint arXiv:1806.00064

    2018 arXiv

  17. [25]

    Curtis Northcutt, Shengxin Zha, Steven Lovegrove, and Richard Newcombe. 2020. Egocom: A multi-person multi-modal egocentric communications dataset. IEEE Transactions on Pattern Analysis and Machine Intelligence

    2020

  18. [26]

    Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5206--5210. IEEE

    2015

  19. [27]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9

    2019

  20. [28]

    Harvey Sacks, Emanuel A Schegloff, and Gail Jefferson. 1974. A simplest systematics for the organization of turn-taking for conversation. language, 50(4):696--735

    1974

  21. [29]

    Harvey Sacks, Emanuel A Schegloff, and Gail Jefferson. 1978. A simplest systematics for the organization of turn taking for conversation. In Studies in the organization of conversational interaction, pages 7--55. Elsevier

    1978

  22. [30]

    Kotaro Shukuri, Ryoma Ishigaki, Jundai Suzuki, Tsubasa Naganuma, Takuma Fujimoto, Daisuke Kawakubo, Masaki Shuzo, and Eisaku Maeda. 2023. Meta-control of dialogue systems using large language models. arXiv preprint arXiv:2312.13715

    2023 arXiv

  23. [31]

    Gabriel Skantze. 2017. Towards a general, continuous model of turn-taking in spoken dialogue using lstm recurrent neural networks. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pages 220--230

    2017

  24. [32]

    Ruijie Tao, Zexu Pan, Rohan Kumar Das, Xinyuan Qian, Mike Zheng Shou, and Haizhou Li. 2021. Is someone speaking? exploring long-term temporal features for audio-visual active speaker detection. In Proceedings of the 29th ACM International Conference on Multimedia, page 3927–3935

    2021

  25. [33]

    Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. 2022. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. Advances in neural information processing systems, 35:10078--10093

    2022

  26. [34]

    Christophe Veaux, Junichi Yamagishi, and Kirsten MacDonald. 2017. Cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit

    2017

  27. [35]

    Bandhav Veluri, Benjamin N Peloquin, Bokai Yu, Hongyu Gong, and Shyamnath Gollakota. 2024. Beyond turn-based interfaces: Synchronous llms as full-duplex dialogue agents. arXiv preprint arXiv:2409.15594

    2024 arXiv

  28. [36]

    Hongji Wang, Chengdong Liang, Shuai Wang, Zhengyang Chen, Binbin Zhang, Xu Xiang, Yanlei Deng, and Yanmin Qian. 2023. Wespeaker: A research and production oriented speaker embedding learning toolkit. In IEEE International Conference on Acoustics, Speech and Signal Processing (...

    2023

  29. [37]

    Jinhan Wang, Long Chen, Aparna Khare, Anirudh Raju, Pranav Dheram, Di He, Minhua Wu, Andreas Stolcke, and Venkatesh Ravichandran. 2024 a . Turn-taking and backchannel prediction with acoustic and large language model fusion. arXiv preprint arXiv:2401.14717

    2024 arXiv

  30. [38]

    Peng Wang, Songshuo Lu, Yaohua Tang, Sijie Yan, Wei Xia, and Yuanjun Xiong. 2024 b . A full-duplex speech dialogue scheme based on large language models. arXiv preprint arXiv:2405.19487

    2024 arXiv

  31. [39]

    Nigel G Ward, Diego Aguirre, Gerardo Cervantes, and Olac Fuentes. 2018. Turn-taking predictions across languages and genres using an lstm recurrent neural network. In 2018 IEEE Spoken Language Technology Workshop (SLT), pages 831--837. IEEE

    2018

  32. [40]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R \'e mi Louf, Morgan Funtowicz, et al. 2019. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771

    2019 arXiv

  33. [41]

    Jiudong Yang, Peiying Wang, Yi Zhu, Mingchao Feng, Meng Chen, and Xiaodong He. 2022. Gated multimodal fusion with contrastive learning for turn-taking prediction in human-robot dialogue. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processi...

    2022

  34. [42]

    Liu Yang, Jieyeon Woo, Catherine Achard, and Catherine Pelachaud. 2023. Exchanging... watch out! arXiv preprint arXiv:2311.04747

    2023 arXiv

  35. [43]

    Victor H Yngve. 1970. On getting a word in edgewise. In Papers from the sixth regional meeting Chicago Linguistic Society, April 16-18, 1970, Chicago Linguistic Society, Chicago, pages 567--578

    1970

  36. [44]

    Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2017. Tensor fusion network for multimodal sentiment analysis. arXiv preprint arXiv:1707.07250

    2017 arXiv

  37. [45]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  38. [46]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.