Pith. sign in

REVIEW 3 major objections 7 minor 43 references

Adapting Speech-Activity Encoders with LoRA Lifts Robot Turn-Taking Prediction

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-09 14:55 UTC pith:AWVX77V3

load-bearing objection Multimodal VAP with VA-related pretrained encoders and LoRA: solid empirical gains but missing ablations on the semantic loss term the 3 major comments →

arxiv 2607.07294 v1 pith:AWVX77V3 submitted 2026-07-08 cs.RO cs.AIcs.CL

Multimodal Voice Activity Projection for Turn-Taking in Social Robots with Voice-Activity-Related Pretrained Encoders

Antonio Cano , Guillermo P\'erez , Luis Merino , Randy Gomez This is my paper
classification cs.RO cs.AIcs.CL
keywords turn-taking predictionvoice activity projectionmultimodal fusionsocial roboticslow-rank adaptationpretrained encodershuman-robot interactionconversation modeling
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that pretrained audio-visual encoders originally built for speech-activity tasks—detecting who is speaking and recognizing audio-visual speech—can be repurposed to predict future conversational turn-taking dynamics in social robot mediation settings. The core mechanism is a Multimodal Voice Activity Projection (MM-VAP) architecture: each speaker's audio and video are encoded independently by a frozen pretrained backbone, the backbone's native cross-modal attention fuses audio and visual signals per speaker, and a second cross-attention layer models how one speaker's state influences the other. Low-Rank Adaptation (LoRA) redirects the pretrained weights toward projecting future voice activity without full fine-tuning. A semantic consistency loss regularizes the 256-state prediction space by grouping microstates that share the same dialogue-level meaning, reducing the influence of rare or ambiguous configurations. The authors argue that encoders pretrained on voice-activity-related tasks transfer more effectively to turn-taking prediction than generic multimodal encoders or hand-crafted visual features, and they test this across five languages and two corpora, reporting improvements on several turn-taking event categories over prior baselines.

Core claim

The central claim is that the representational knowledge captured by encoders pretrained on speech-activity tasks (active speaker detection, audio-visual speech recognition) transfers to the distinct problem of predicting future voice activity patterns for turn-taking, and that this transfer is more effective when the backbone's original inter-modal attention mechanism is preserved and adapted via LoRA rather than replaced by separately trained fusion layers. The semantic consistency loss further improves optimization by aligning the fine-grained 256-state prediction distribution with higher-level dialogue activity patterns, which stabilizes training on an inherently unbalanced state space.

What carries the argument

The architecture has five stages: (1) preprocessing segments synchronized audio-visual streams per speaker; (2) frozen pretrained backbones (TalkNet or WhisperFlamingo) encode each speaker's audio and visual modalities, with LoRA adapters added to selected layers; (3) the backbone's native cross-modal attention fuses audio and visual embeddings per speaker; (4) an inter-speaker cross-attention layer models relational dynamics between the two speakers' representations; (5) two prediction heads output a 256-state voice activity projection distribution and binary voice activity probabilities. Turn-taking events (shift, hold, backchannel prediction, shift prediction) are derived zero-shot by sum

Load-bearing premise

The load-bearing assumption is that encoders pretrained on speech-activity tasks capture temporal and multimodal dynamics that transfer to predicting future turn-taking events. Active speaker detection and audio-visual speech recognition do not explicitly model conversational turn structure, so the paper assumes their learned representations contain enough predictive signal about who will speak next that LoRA adaptation and inter-speaker attention can recover it. The paper's

What would settle it

The key empirical claim would be falsified if replacing the speech-activity-pretrained backbones with equally sized generic multimodal encoders (not pretrained on speech-related tasks) adapted with the same LoRA strategy and inter-speaker attention yielded comparable or superior turn-taking event prediction performance. The paper includes one such comparison (CPC+3DResNet as a non-pretrained baseline), but a stronger falsification would use a generic multimodal backbone with native cross-modal attention to isolate whether the gains come from speech-activity-specific pretraining or from the

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If speech-activity-pretrained encoders transfer to turn-taking prediction, the same principle may extend to other conversational prediction tasks—e.g., next-speaker selection in multi-party dialogue or backchannel timing—by repurposing encoders from adjacent speech tasks rather than training task-specific fusion models from scratch.
  • The LoRA-adaptation strategy suggests that social robots with limited compute could deploy large pretrained multimodal encoders for real-time turn-taking by updating only low-rank adapter weights, though the paper notes real-time validation remains future work.
  • The semantic consistency loss pattern—grouping fine-grained output states by semantic equivalence—could generalize to other self-supervised prediction problems with large, unbalanced discrete output spaces where many microstates map to the same downstream decision.
  • If the gains stem primarily from the backbone's native cross-modal attention rather than the specific pretraining task, then any multimodal backbone with strong internal audio-visual fusion could serve as a turn-taking encoder, broadening the design space beyond speech-activity-specific models.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper does not isolate whether the gains come from the speech-activity relevance of the pretraining task or from the architectural property of having a native cross-modal attention mechanism. If the latter dominates, the choice of TalkNet/WhisperFlamingo over other fusion-capable backbones may be less critical than the paper implies.
  • The semantic consistency loss effectively trades resolution for stability in the 256-state space, but the paper does not test whether a smaller, directly semantic state space would perform comparably without the two-stage collapse, which could simplify the architecture.
  • The evaluation on the Haru EDR corpus uses only English-trained models on English data; the cross-lingual transfer to a robot mediation scenario in other languages remains untested, and the lower Japanese performance suggests language-specific adaptation may be necessary for deployment in non-English-speaking robot mediation contexts.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. This paper proposes a Multimodal Voice Activity Projection (MM-VAP) framework for turn-taking prediction in social robotics. The approach extends the audio-only VAP formulation to audio-visual inputs by integrating pretrained backbones (TalkNet, WhisperFlamingo) originally trained on speech-activity-related tasks (ASD, AVSR), adapting them via LoRA. An inter-speaker attention stage models relational dynamics between participants, and a semantic consistency loss regularizes the 256-state output space. Experiments on NoXi, NoXi+J, and the Haru EDR corpus show improvements over baselines on several turn-taking events, particularly shift-prediction and shift/hold. The work is motivated by Haru's role as a social mediator in human-human interaction.

Significance. The paper addresses a practically important problem for social robotics: predictive rather than reactive turn-taking. The central design choice—using pretrained encoders whose original tasks are voice-activity-related (ASD, AVSR) rather than generic multimodal encoders—is reasonable and the empirical evidence supports it. The release of source code and pretrained models is a positive step for reproducibility. The multilingual evaluation across five languages and the additional validation on a robot-specific corpus (Haru EDR) broaden the evidence base beyond typical single-corpus studies. The application of MM-VAP in a robotics mediator context is, to my knowledge, novel.

major comments (3)
  1. Section I and Section III.E: The semantic consistency loss (L_sem, Eq. 3) is presented as a core contribution, but no ablation isolates its effect. Table II compares pretrained backbones against a non-pretrained baseline (CPC+3DResNet), but all pretrained variants include L_sem, so the reader cannot determine whether gains stem from the pretrained encoders, LoRA adaptation, inter-speaker attention, or L_sem itself. Since L_sem is listed as a distinct contribution alongside backbone selection, an ablation (with and without L_sem, holding the backbone fixed) is needed to justify its inclusion as a contribution. Without it, the novelty claim narrows to applying existing pretrained encoders with LoRA to VAP.
  2. Table III and the Discussion (Section V): The comparison with prior work is partially non-comparable. The first row (Russell et al.) uses different event definitions (Keep/Hold, Turn/Shift, marked with daggers) and reports only S/H and BC-pred, while Saga et al. report only S/H. The claim of 'improvements over the current baselines' is strongest for S-pred and S/L, but for BC-pred the proposed method underperforms Onishi et al. on several subsets (e.g., English: 0.45 vs 0.79; German: 0.43 vs 0.73; Japanese: 0.52 vs 0.48 is comparable but All: 0.45 vs 0.71). The manuscript should explicitly acknowledge where prior methods still outperform the proposed approach and clarify which comparisons are like-for-like.
  3. The inter-speaker attention stage (Section III.B, z = Phi_inter-spkrs(H^(1), H^(2))) is described as modeling relational dynamics, but no ablation or analysis isolates its contribution from the backbone's own inter-modal attention. Since TalkNet and WhisperFlamingo already contain cross-attention between audio and visual modalities, the marginal value of the additional inter-speaker attention is unclear. A simple ablation (removing this stage or replacing it with concatenation) would clarify whether this architectural choice is load-bearing for the gains.
minor comments (7)
  1. Section III.C: The LoRA hyperparameters differ between backbones (alpha=16, r=16 for TalkNet; alpha=64, r=32 for WhisperFlamingo) with the justification 'seeking deeper adaptation.' The rationale for these specific values is not explained beyond this. A brief justification or citation would strengthen reproducibility.
  2. Section III.E: The semantic loss coefficient mu is described as 'empirically chosen' in [0.1, 0.2], but the exact value used in each experiment is not reported. This should be specified.
  3. Table I: BC-pred F1 scores are notably low across all languages (0.41-0.77). The paper does not discuss why backchannel prediction remains difficult or whether the semantic consistency loss helps or hurts this specific event. A brief discussion would be valuable.
  4. Section IV.A: The context window (10s) and prediction horizon (2s) are stated but not justified relative to prior VAP work. A sentence comparing these choices to the original VAP settings would help readers assess their appropriateness.
  5. Table III: The footnote markers (dagger, double-dagger) are explained, but the table layout makes it slightly hard to parse which rows correspond to which method. Clarifying the grouping would help.
  6. Section II: The related work mentions that [20] used 'generic' pretrained encoders not related to voice activity, but does not include an experimental comparison against [20]'s approach. A brief discussion of why a direct comparison was not feasible would contextualize the contribution.
  7. Fig. 2: The architecture diagram is informative but the text in the bottom-right panel is small. A higher-resolution or simplified version would aid understanding.

Circularity Check

0 steps flagged

No circularity found: training targets (future VA states) are distinct from evaluation metrics (turn-taking events), and the semantic consistency loss regularizes based on current-activity groupings rather than the prediction target itself

full rationale

The paper's derivation chain is self-contained against external benchmarks. The self-supervised training objective predicts future voice activity (256-state distribution) from VA annotations, while turn-taking events are derived zero-shot by summing microstate probabilities — the training signal is structurally distinct from the evaluation metric. The semantic consistency loss (Eq. 3) groups microstates by current speaker activity patterns derived from VA ground truth, not from the predicted events, so it does not define the prediction in terms of itself. Pretrained backbones (TalkNet [29], WhisperFlamingo [30]) and the VAP formulation [18] are all external work by other authors. The sole self-citation [3] (Cano & Merino) provides motivational context only and is not load-bearing for any technical claim. No step reduces by construction to its inputs.

Axiom & Free-Parameter Ledger

6 free parameters · 4 axioms · 0 invented entities

The paper introduces no new physical entities or mathematical objects beyond standard neural network components. The free parameters are LoRA hyperparameters and loss coefficients, all empirically chosen. The axioms are inherited from the VAP framework or are domain assumptions about representation transfer.

free parameters (6)
  • LoRA rank r = 16 (TalkNet), 32 (WhisperFlamingo)
    Chosen empirically based on backbone performance; controls adaptation flexibility.
  • LoRA scaling alpha = 16 (TalkNet), 64 (WhisperFlamingo)
    Chosen empirically; determines adaptation strength.
  • Semantic loss coefficient mu = 0.1-0.2
    Empirically chosen range; controls the semantic consistency loss contribution.
  • Learning rate = 0.009
    Initial value with linear scheduler; chosen for stable optimization.
  • Projection window bins = 4 bins per speaker (200, 400, 600, 800 ms)
    Inherited from original VAP; defines the 256-state output space.
  • Context window length = 10s with 0.5s slide
    Chosen for experimental setup; not ablated.
axioms (4)
  • domain assumption Turn-taking can be modeled as future voice activity projection over a finite temporal horizon.
    Inherited from the original VAP formulation (Ekstedt & Skantze 2022); the entire framework depends on this equivalence.
  • domain assumption Pretrained encoders optimized for speech-related tasks (ASD, AVSR) provide representations suitable for VA projection.
    Stated in Section III-C as the core hypothesis; the backbone selection criteria assume this transfer is valid.
  • standard math The 256-state VAP space can be collapsed into four speaker-activity bins for zero-shot event inference.
    Section III-D; the bin-level aggregation preserves the VAP state distribution by construction.
  • domain assumption Self-supervised learning from VA labels suffices for turn-taking event prediction without manual event annotations.
    Inherited from original VAP; the zero-shot inference approach depends on this.

pith-pipeline@v1.1.0-glm · 17286 in / 2585 out tokens · 279677 ms · 2026-07-09T14:55:02.203216+00:00 · methodology

0 comments
read the original abstract

Turn-taking prediction is a key requirement for social robots involved in human-human interaction, particularly in mediator settings, where the robot must anticipate conversational dynamics rather than merely react to pauses. This work presents a Multimodal Voice Activity Projection (MM-VAP) framework that extends the original audio-only VAP formulation to synchronized audio-visual inputs while preserving its self-supervised future-projection objective. The proposed approach builds on pretrained audio-visual backbones originally optimized for speech-related tasks and adapts them through Low-Rank Adaptation to the multimodal turn-taking problem. After independent speaker encoding, an inter-speaker attention stage models the relational dynamics required to project future voice activity. In addition, a semantic consistency loss is introduced to regularize the 256-state output space according to higher-level dialogue activity patterns. Experiments on NoXi and NoXi+J showed improvements over the current baselines, particularly for some turn-taking events. Additional evaluation on the Haru EDR corpus further supported the suitability of this direction for mediation-oriented human-robot interaction.

Figures

Figures reproduced from arXiv: 2607.07294 by Antonio Cano, Guillermo P\'erez, Luis Merino, Randy Gomez.

Figure 1
Figure 1. Figure 1: Haru operating as social mediator in two-party [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Proposed MM-VAP architecture for dyadic interaction overview. Blue and red paths indicate the flow from raw inputs [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

43 extracted references · 43 canonical work pages · 10 internal anchors

  1. [1]

    A Simplest Systematics for the Organization of Turn-Taking for Conversation,

    H. Sacks, E. A. Schegloff, and G. Jefferson, “A Simplest Systematics for the Organization of Turn-Taking for Conversation,”Language, vol. 50, p. 696, Dec. 1974

    work page 1974

  2. [2]

    Palinko, L

    O. Palinko, L. Bodenhagen, J.-J. Cabibihan, K. Fischer, S. ˇSabanovi´c, K. Winkle, L. Behera, S. S. Ge, D. Chrysostomou, W. Jiang, and H. He, eds.,Social Robotics: 16th International Conference, ICSR + AI 2024, Odense, Denmark, October 23–26, 2024, Proceedings, Part III, vol. 15563 ofLecture Notes in Computer Science. Singapore: Springer Nature Singapore,...

    work page 2024

  3. [3]

    Towards improving turn-taking in social robots using Visual-Only V oice Activity Detection in multimodal dialogue systems,

    A. Cano and L. Merino, “Towards improving turn-taking in social robots using Visual-Only V oice Activity Detection in multimodal dialogue systems,”

    work page

  4. [4]

    Design of Social Features for Robot-mediated Cross-cultural Interaction,

    S. Cooper, R. Gomez, D. Szapiro, and L. Merino, “Design of Social Features for Robot-mediated Cross-cultural Interaction,” inCompan- ion of the 2024 ACM/IEEE International Conference on Human-Robot Interaction, (Boulder CO USA), pp. 351–355, ACM, Mar. 2024

    work page 2024

  5. [5]

    Haru in the Care Network: Stakeholder Perspec- tives on Privacy with Social Robots in Pediatrics,

    L. Levinson, G. Alvarez-Benito, J. G. Amores, D. Szapiro, R. Gomez, and S. ˇSabanovi´c, “Haru in the Care Network: Stakeholder Perspec- tives on Privacy with Social Robots in Pediatrics,”Proceedings of the ACM on Human-Computer Interaction, vol. 9, pp. 1–28, Oct. 2025

    work page 2025

  6. [6]

    Building Friendships Across Borders: The Role of Social Robot Haru in Children Group Communication and Con- nection Development,

    Z. Yi, L. Levinson, D. Delgado-Chaves, J. M. Perez-Moleron, N. Bougria, A. Krummheuer, M. Rehm, A. K. Møller, K. K. Rams- gaard, S. Auala, H. Winschiers-Theophilus, E. Nepolo, D. Calero, D. Dal Moro, D. Serrano, M. Dalmau-Moreno, R. Gomez, L. Merino, and S. ˇSabanovi´c, “Building Friendships Across Borders: The Role of Social Robot Haru in Children Group ...

    work page 2025

  7. [7]

    Multimodal Transformer Models for Turn-Taking Prediction: Effects on Conversational Dynamics of Human-Agent Interaction During Cooperative Gameplay,

    Y .-H. Bae and C. C. Bennett, “Multimodal Transformer Models for Turn-Taking Prediction: Effects on Conversational Dynamics of Human-Agent Interaction During Cooperative Gameplay,” 2025

    work page 2025

  8. [8]

    When and How to Express Empathy in Human-Robot Interaction Scenarios,

    C. A. Cruz, E. C. Montiel-Vazquez, C. Maeda, and R. Gomez, “When and How to Express Empathy in Human-Robot Interaction Scenarios,” in2025 34th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), (Eindhoven, Nether- lands), pp. 1070–1077, IEEE, Aug. 2025

    work page 2025

  9. [9]

    Visual Cues Enhance Predic- tive Turn-Taking for Two-Party Human Interaction,

    S. O. Russell and N. Harte, “Visual Cues Enhance Predic- tive Turn-Taking for Two-Party Human Interaction,” May 2025. arXiv:2505.21043 [cs]

    work page arXiv 2025

  10. [10]

    Turn-taking in Conversational Systems and Human- Robot Interaction: A Review,

    G. Skantze, “Turn-taking in Conversational Systems and Human- Robot Interaction: A Review,”Computer Speech & Language, vol. 67, p. 101178, May 2021

    work page 2021

  11. [11]

    Data-driven models for timing feedback responses in a Map Task dialogue system,

    R. Meena, G. Skantze, and J. Gustafson, “Data-driven models for timing feedback responses in a Map Task dialogue system,”Computer Speech & Language, vol. 28, pp. 903–922, July 2014

    work page 2014

  12. [12]

    On temporal aspects of turn taking in conversational dialogues,

    L. T. Bosch, N. Oostdijk, and L. Boves, “On temporal aspects of turn taking in conversational dialogues,”Speech Communication, vol. 47, pp. 80–86, Sept. 2005

    work page 2005

  13. [13]

    Turn-Taking Modelling in Conversational Systems: A Review of Recent Advances,

    R. A. Patamia, H. P. T. Dinh, M. Liu, and A. Cosgun, “Turn-Taking Modelling in Conversational Systems: A Review of Recent Advances,” Technologies, vol. 13, p. 591, Dec. 2025

    work page 2025

  14. [14]

    Pauses, gaps and overlaps in conversa- tions,

    M. Heldner and J. Edlund, “Pauses, gaps and overlaps in conversa- tions,”Journal of Phonetics, vol. 38, pp. 555–568, Oct. 2010

    work page 2010

  15. [15]

    Timing in turn-taking and its impli- cations for processing models of language,

    S. C. Levinson and F. Torreira, “Timing in turn-taking and its impli- cations for processing models of language,”Frontiers in Psychology, vol. 6, June 2015

    work page 2015

  16. [16]

    Timed picture naming in seven languages,

    E. Bates, S. D’Amico, T. Jacobsen, A. Sz ´ekely, E. Andonova, A. De- vescovi, D. Herron, C. Ching Lu, T. Pechmann, C. Pl ´eh, N. Wicha, K. Federmeier, I. Gerdjikova, G. Gutierrez, D. Hung, J. Hsu, G. Iyer, K. Kohnert, T. Mehotcheva, A. Orozco-Figueroa, A. Tzeng, and O. Tzeng, “Timed picture naming in seven languages,”Psychonomic Bulletin & Review, vol. 10...

    work page 2003

  17. [17]

    Multimodal Turn Analysis and Prediction for Multi-party Conversations,

    M.-C. Lee, M. Trinh, and Z. Deng, “Multimodal Turn Analysis and Prediction for Multi-party Conversations,” inINTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION, (Paris France), pp. 436–444, ACM, Oct. 2023

    work page 2023

  18. [18]

    Voice Activity Projection: Self-supervised Learning of Turn-taking Events

    E. Ekstedt and G. Skantze, “V oice Activity Projection: Self-supervised Learning of Turn-taking Events,” May 2022. arXiv:2205.09812 [eess]

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  19. [19]

    Multimodal V oice Activity Projection for Turn-Taking and Effects on Speaker Adaptation,

    K. Onishi, H. Tanaka, and S. Nakamura, “Multimodal V oice Activity Projection for Turn-Taking and Effects on Speaker Adaptation,”IEICE Transactions on Information and Systems, vol. E108.D, pp. 445–453, June 2025

    work page 2025

  20. [20]

    Voice Activity Projection Model with Multimodal Encoders

    T. Saga and C. Pelachaud, “V oice Activity Projection Model with Multimodal Encoders,” June 2025. arXiv:2506.03980 [cs]

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  21. [21]

    Predicting Turn-Taking and Backchannel in Human-Machine Conversations Using Linguistic, Acoustic, and Visual Signals

    Y . Lin, Y . Zheng, M. Zeng, and W. Shi, “Predicting Turn-Taking and Backchannel in Human-Machine Conversations Using Linguistic, Acoustic, and Visual Signals,” May 2025. arXiv:2505.12654 [cs]

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  22. [22]

    Behind the Scenes: Mechanistic Interpretability of LoRA-adapted Whisper for Speech Emotion Recognition,

    Y . Ma, X. Lu, J. Sang, X. Jiang, and R. Li, “Behind the Scenes: Mechanistic Interpretability of LoRA-adapted Whisper for Speech Emotion Recognition,” Jan. 2026. arXiv:2509.08454 [cs]

    work page arXiv 2026

  23. [23]

    GRPO- Guided Modality Selection Enhanced LoRA-Tuned LLMs for Multi- modal Emotion Recognition,

    Y . Chen, S. Yang, Y . Xiang, R. Song, Y . Huang, and Z. Yu, “GRPO- Guided Modality Selection Enhanced LoRA-Tuned LLMs for Multi- modal Emotion Recognition,”

    work page

  24. [24]

    LoRA- Whisper: Parameter-Efficient and Extensible Multilingual ASR,

    Z. Song, J. Zhuo, Y . Yang, Z. Ma, S. Zhang, and X. Chen, “LoRA- Whisper: Parameter-Efficient and Extensible Multilingual ASR,” in Interspeech 2024, pp. 3934–3938, ISCA, Sept. 2024

    work page 2024

  25. [25]

    M3SD: Multi-modal, Multi-scenario and Multi-language Speaker Diarization Dataset

    S. Wu, “M3SD: Multi-modal, Multi-scenario and Multi-language Speaker Diarization Dataset,” June 2025. arXiv:2506.14427 [eess]

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  26. [26]

    Multimodal Large Language Model with LoRA Fine-Tuning for Multimodal Sentiment Analysis,

    J. Mu, W. Wang, W. Liu, T. Yan, and G. Wang, “Multimodal Large Language Model with LoRA Fine-Tuning for Multimodal Sentiment Analysis,”ACM Transactions on Intelligent Systems and Technology, vol. 16, pp. 1–23, Dec. 2025

    work page 2025

  27. [27]

    Multimodal Large Language Models with Fusion Low Rank Adaptation for Device Directed Speech Detection

    S. Palaskar, O. Rudovic, S. Dharur, F. Pesce, G. Krishna, A. Sivara- man, J. Berkowitz, A. H. Abdelaziz, S. Adya, and A. Tewfik, “Mul- timodal Large Language Models with Fusion Low Rank Adaptation for Device Directed Speech Detection,” June 2024. arXiv:2406.09617 [cs]

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  28. [28]

    Robust Adaptation of Large Multimodal Models for Retrieval Augmented Hateful Meme Detection,

    J. Mei, J. Chen, G. Yang, W. Lin, and B. Byrne, “Robust Adaptation of Large Multimodal Models for Retrieval Augmented Hateful Meme Detection,”

    work page

  29. [29]

    Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection,

    R. Tao, Z. Pan, R. K. Das, X. Qian, M. Z. Shou, and H. Li, “Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection,” inProceedings of the 29th ACM International Conference on Multimedia, pp. 3927–3935, Oct

    work page

  30. [30]

    arXiv:2107.06592 [eess]

    work page internal anchor Pith review Pith/arXiv arXiv

  31. [31]

    Whisper-Flamingo: Integrating Visual Features into Whisper for Audio-Visual Speech Recognition and Translation

    A. Rouditchenko, Y . Gong, S. Thomas, L. Karlinsky, H. Kuehne, R. Feris, and J. Glass, “Whisper-Flamingo: Integrating Visual Features into Whisper for Audio-Visual Speech Recognition and Translation,” Nov. 2024. arXiv:2406.10082 [eess]

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  32. [32]

    Triadic Multi- party V oice Activity Projection for Turn-taking in Spoken Dia- logue Systems,

    M. Elmers, K. Inoue, D. Lala, and T. Kawahara, “Triadic Multi- party V oice Activity Projection for Turn-taking in Spoken Dia- logue Systems,” inInterspeech 2025, pp. 3015–3019, Aug. 2025. arXiv:2507.07518 [cs]

    work page arXiv 2025

  33. [33]

    Yeah, Un, Oh: Continuous and Real-time Backchannel Prediction with Fine-tuning of V oice Activity Projection,

    K. Inoue, D. Lala, G. Skantze, and T. Kawahara, “Yeah, Un, Oh: Continuous and Real-time Backchannel Prediction with Fine-tuning of V oice Activity Projection,”

    work page

  34. [34]

    Predicting End-of- turn and Backchannel Based on Multimodal V oice Activity Prediction Model,

    R. Ishii, S. Eitoku, R. Yokoyama, and J. Sawase, “Predicting End-of- turn and Backchannel Based on Multimodal V oice Activity Prediction Model,” inProceedings of the 27th International Conference on Multimodal Interaction, (Canberra Australia), pp. 446–455, ACM, Oct. 2025

    work page 2025

  35. [35]

    Multi- lingual Turn-taking Prediction Using V oice Activity Projection,

    K. Inoue, B. Jiang, E. Ekstedt, T. Kawahara, and G. Skantze, “Multi- lingual Turn-taking Prediction Using V oice Activity Projection,” Mar

    work page

  36. [36]

    arXiv:2403.06487 [cs]

    work page internal anchor Pith review Pith/arXiv arXiv

  37. [37]

    Investigating the Language Independence of V oice Activity Projection Models through Stan- dardization of Speech Segmentation Labels,

    Y . Sato, Y . Chiba, and R. Higashinaka, “Investigating the Language Independence of V oice Activity Projection Models through Stan- dardization of Speech Segmentation Labels,” in2024 Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), (Macau, Macao), pp. 1–6, IEEE, Dec. 2024

    work page 2024

  38. [38]

    Real-time and Continuous Turn-taking Prediction Using Voice Activity Projection

    K. Inoue, B. Jiang, E. Ekstedt, T. Kawahara, and G. Skantze, “Real- time and Continuous Turn-taking Prediction Using V oice Activity Projection,” Jan. 2024. arXiv:2401.04868 [cs]

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  39. [39]

    Applying General Turn-Taking Models to Conversational Human-Robot Interaction,

    G. Skantze and B. Irfan, “Applying General Turn-Taking Models to Conversational Human-Robot Interaction,” in2025 20th ACM/IEEE International Conference on Human-Robot Interaction (HRI), (Mel- bourne, Australia), pp. 859–868, IEEE, Mar. 2025

    work page 2025

  40. [40]

    A Noise-Robust Turn-Taking System for Real-World Dialogue Robots: A Field Experiment,

    K. Inoue, Y . Okafuji, J. Baba, Y . Ohira, K. Hyodo, and T. Kawa- hara, “A Noise-Robust Turn-Taking System for Real-World Dialogue Robots: A Field Experiment,” in2025 IEEE/RSJ International Con- ference on Intelligent Robots and Systems (IROS), (Hangzhou, China), pp. 874–879, IEEE, Oct. 2025

    work page 2025

  41. [41]

    Multimodal V oice Activity Prediction: Turn-taking Events Detection in Expert-Novice Conver- sation,

    K. Onishi, H. Tanaka, and S. Nakamura, “Multimodal V oice Activity Prediction: Turn-taking Events Detection in Expert-Novice Conver- sation,” inInternational Conference on Human-Agent Interaction, (Gothenburg Sweden), pp. 13–21, ACM, Dec. 2023

    work page 2023

  42. [42]

    The NoXi database: multimodal recordings of mediated novice-expert interactions,

    A. Cafaro, J. Wagner, T. Baur, S. Dermouche, M. Torres Torres, C. Pelachaud, E. Andr ´e, and M. Valstar, “The NoXi database: multimodal recordings of mediated novice-expert interactions,” in Proceedings of the 19th ACM International Conference on Multimodal Interaction, (Glasgow UK), pp. 350–359, ACM, Nov. 2017

    work page 2017

  43. [43]

    Multilingual Dyadic Interaction Corpus NoXi+J: Toward Understanding Asian-European Non-verbal Cultural Characteristics and their Influences on Engagement

    M. Funk, S. Okada, and E. Andr ´e, “Multilingual Dyadic Interaction Corpus NoXi+J: Toward Understanding Asian-European Non-verbal Cultural Characteristics and their Influences on Engagement,” in International Conference on Multimodel Interaction, pp. 224–233, Nov. 2024. arXiv:2409.13726 [cs]. Accepted preprint. This manuscript has been accepted for presen...

    work page internal anchor Pith review Pith/arXiv arXiv 2024