A multi-modal (text, audio, video) model trained on a newly collected 210-hour conversation dataset predicts turn-taking and backchannel actions with F1 about 0.81 and 0.91.
Exchanging... Watch out!
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
During a conversation, individuals take turns speaking and engage in exchanges, which can occur smoothly or involve interruptions. Listeners have various ways of participating, such as displaying backchannels, signalling the aim to take a turn, waiting for the speaker to yield the floor, or even interrupting and taking over the conversation. These exchanges are commonplace in natural interactions. To create realistic and engaging interactions between human participants and embodied conversational agents (ECAs), it is crucial to equip virtual agents with the ability to manage these exchanges. This includes being able to initiate or respond to signals from the human user. In order to achieve this, we annotate, analyze and characterize these exchanges in human-human conversations. In this paper, we present an analysis of multimodal features, with a focus on prosodic features such as pitch (F0) and loudness, as well as facial expressions, to describe different types of exchanges.
fields
cs.CL 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Predicting Turn-Taking and Backchannel in Human-Machine Conversations Using Linguistic, Acoustic, and Visual Signals
A multi-modal (text, audio, video) model trained on a newly collected 210-hour conversation dataset predicts turn-taking and backchannel actions with F1 about 0.81 and 0.91.