{"id":"3bd49c46-c5c9-4fdd-97b5-377c56bd1533","arxiv_id":"2505.12654","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A multi-modal (text, audio, video) model trained on a newly collected 210-hour conversation dataset predicts turn-taking and backchannel actions with F1 about 0.81 and 0.91.","lead":"This paper builds a large video dataset of two-person face-to-face conversations and a model that predicts when a listener should keep listening, take the turn, or say a short backchannel such as 'hmm'. It reports large F1 gains over reimplemented baselines on this new dataset.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA claim rests on automatic TURN/BACKCHANNEL labels; the manual validation in §A.3 checks framing, audio, and subtitles but never validates those labels, so the asserted >95% label precision is unsubstantiated.","rationale":"The reader's weakest-assumption is the right load-bearing concern: the evaluation measures an automatically derived proxy. This pass sharpens it by showing an internal gap: the manual validation described in §A.3 never asks validators about the actual TURN/BACKCHANNEL labels, so the '>95% precision' assertion in §3.1.5 is not supported by the manuscript's own evidence. The public dataset and code release are real strengths, and the ablations are internally consistent, but the label-validity issue determines what the F1 numbers mean at all. The proposed audit is feasible with the released data and would settle whether the SOTA statement needs to be qualified. Since the reader already issued a conditional verdict, this concern does not change the verdict; it strengthens the specific condition that should be imposed before the headline claims are taken at face value.","tokens_in":15005,"tokens_out":9734,"duration_ms":108417,"concrete_test":"Sample roughly 500 clips from MM-F2F and run a label-validity audit. Two trained annotators independently mark, for each clip, whether the final word of the WhisperX segment is a genuine transition-relevance place (and, using the paper's speaker/ASD assignments, whether the next spoken segment is by the other speaker), and whether each vocabulary-matched token actually functions as a backchannel. Compute Cohen's kappa and the precision/recall of the pipeline labels against these judgments. If kappa is below 0.7, or if many TURN labels are not followed by a speaker change, the reported SOTA should be re-described as proxy-task performance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central SOTA claim is only meaningful if the labels correspond to turn-taking and backchannel behavior. In §3.1.5, TURN is defined as the last word of each WhisperX sentence-level ASR segment, and BACKCHANNEL as any word matching a fixed vocabulary from prior work. WhisperX segments are determined by silence and punctuation, not by conversational turn structure, so a mid-utterance pause can create a spurious TURN, and a fluent speaker change without a pause can be missed. The manual 'double-checking' in §A.3 asks validators only about frame composition, active-speaker boxes, subtitle alignment, and naturalness (Q1–Q4); no question checks whether the TURN or BACKCHANNEL label is correct. Therefore §3.1.5's claim of better than 95% precision for the automatic annotations is unsupported for the target labels, and no evidence links the proxies to natural behavior. Because all F1 gains (turn 0.811 vs 0.739; backchannel 0.906 vs 0.680) are computed on these labels, the abstract's 10%/33% SOTA improvement may describe detection of ASR segment boundaries plus a word list rather than the conversational phenomena named in the title. The backchannel vocabulary is not listed, making the annotation rule not fully reproducible.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces MM-F2F, a 210-hour, 1.5M-word English dyadic video dataset with word-level KEEP/TURN/BACKCHANNEL labels produced by an automatic pipeline (WhisperX ASR, speaker clustering, TalkNet active speaker detection, and rule-based annotation: TURN is the last word of an ASR sentence, BACKCHANNEL is a match against a fixed vocabulary). It then proposes a tri-modal turn-taking/backchannel prediction framework with a low-rank flexible fusion module and random modality dropout, supporting any combination of text, audio, and video inputs. On the authors' own test split, the tri-modal model reports F1 = 0.811 for TURN and 0.906 for BACKCHANNEL, exceeding the reimplemented baselines and yielding the abstract's claimed 10% and 33% relative F1 improvements.","tokens_in":15237,"tokens_out":6321,"duration_ms":65853,"significance":"The dataset is potentially a large-scale resource: it is an order of magnitude larger than EgoCom and is the first dyadic face-to-face corpus with text, audio, video, and backchannel annotations; the authors release code and dataset and include a useful de-identification study (Appendix A.1). The fusion design is clean, and the random modality dropout result (Table 6) demonstrates a practical robustness benefit for missing modalities. However, the significance of the state-of-the-art claim depends entirely on the validity of the automatically derived TURN/BACKCHANNEL labels; the current manual validation does not verify those labels, and all comparisons are point estimates on an internal benchmark. The work is therefore a solid engineering contribution whose headline empirical claim is not yet established.","major_comments":[{"comment":"The annotation rule in §3.1.5 defines TURN as the last word of each WhisperX sentence-level ASR segment and BACKCHANNEL by matching an unlisted fixed vocabulary. Because WhisperX segments on silence and punctuation rather than conversational turn structure, a mid-utterance pause creates a spurious TURN and a fluent speaker change without a pause is missed. The manual double-check in §A.3 asks validators only about frame composition, active-speaker boxes, subtitle accuracy, and naturalness (Q1-Q4); no question checks whether the TURN or BACKCHANNEL label itself is correct. Consequently the §3.1.5 claim of better than 95% precision is unsupported for the labels actually used in training and evaluation, and the reported gains (e.g., Turn F1 0.811 vs 0.739 and BC F1 0.906 vs 0.680 in Table 5) may largely measure prediction of ASR segment boundaries plus a word list rather than the conversational phenomena named in the title. I request a sample-based human agreement study on the TURN/BACKCHANNEL labels themselves (e.g., several hundred words annotated by independent raters against the video) and publication of the backchannel vocabulary.","section":"§3.1.5 and §A.3"},{"comment":"All reported results are single point estimates on the authors' own test split, with no standard deviations across random seeds, no confidence intervals, and no significance tests. The baselines in Table 5 are reimplemented by the authors on the same automatic labels, so the state-of-the-art claim is relative to an internal, self-constructed benchmark; without variance estimates or an evaluation on an independently annotated corpus (e.g., a hand-labeled subset of MM-F2F or an existing public corpus such as EgoCom or FTAD), the 10%/33% improvement of the abstract is not distinguishable from training noise or from artifacts of the reimplementation. Please report mean +/- standard deviation over at least three seeds and, if feasible, validate on externally labeled data.","section":"§5.4 and Tables 3-6"},{"comment":"The limitation section acknowledges a failure case in which a speaker pauses with semantically incomplete content and the model predicts turn-taking instead of backchannel. Under the §3.1.5 definition, however, a pause is exactly what creates a TURN label, so this failure case is not merely a model deficiency but a direct consequence of the label proxy. The section should state this explicitly, or provide evidence that WhisperX sentence boundaries align with genuine turn transitions in these cases.","section":"§7 and Fig. 5"}],"minor_comments":[{"comment":"The symbols I_T, I_A, I_V are described as indicator functions but are used as identity-or-constant-one mappings; please rename them to avoid confusion with set indicators.","section":"Eq. (4)"},{"comment":"Section 3.1.1 says conversations containing personal information were manually removed, while Appendix A.2 says text dialogue content is de-identified; please clarify whether the released transcripts are original or rewritten.","section":"§3.1.1 vs §A.2"},{"comment":"The caption 'GT. Pred.' suggests a comparison, but no predicted labels or probability values are shown in the figure; please clarify what is being visualized.","section":"Fig. 5"},{"comment":"For single-modality input the paper says the model defaults to the uni-modal encoder head, but Eq. (4) with one active modality would also produce a fused representation; please explain why the uni-modal head is preferred.","section":"§4.3.2"},{"comment":"The w/o RMDT bi-modal rows collapse to near-zero F1 for some classes (e.g., T+V Turn F1 = 0.001 and BC F1 = 0.005), which suggests degenerate prediction; please report class distributions or confusion matrices for these rows.","section":"Table 6"},{"comment":"The abstract's '10% increase in F1-score on turn-taking and a 33% increase on backchannel prediction' are relative improvements over the Wang et al. baseline from Table 5; please state this explicitly.","section":"Abstract"},{"comment":"The related-work section mentions 'Chang et al.' and 'Yang et al.' without reference numbers; please add the corresponding citations.","section":"Related Work"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the central claim is contingent on label validity. I would not reject outright because the released dataset and the flexible fusion framework are useful contributions, but the paper should not claim state-of-the-art performance without human validation of the TURN/BACKCHANNEL labels and without variance estimates. If those are supplied, I would be inclined to accept."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Useful resource, reasonable baseline, but do not quote the SOTA numbers without reading the label definitions.\n\nWhat is genuinely new: MM-F2F, 210 hours / 1.5M words of de-identified dyadic conversation video with word-level TURN/BACKCHANNEL/KEEP annotations. That is a real contribution for full-duplex dialogue research, and the de-identification study in A.1 (four train/test settings) is a thoughtful check that synthetic faces and perturbed voiceprints do not destroy behavior-relevant signal. The fusion module is a modest extension of LMF with modality selection and random dropout, but Table 6 shows the dropout scheme does real work when modalities are missing at inference. The internal ablations are consistent, and the writing is honest about the failure case in Fig. 5.\n\nThe soft spots are not minor. The label definition in 3.1.5 makes TURN the last word of each WhisperX sentence-level segment and BACKCHANNEL a match against a fixed vocabulary from prior work. WhisperX segments by silence and punctuation, not by conversational turn structure, so a pause mid-utterance creates a spurious TURN and a fluent speaker change without a pause is missed. The appendix manual double-check (A.3) asks about frame composition, active speaker boxes, subtitle alignment, and naturalness; none of Q1–Q4 checks whether the TURN or BACKCHANNEL label is right. So the claimed >95% annotation precision is unsubstantiated for the target labels, and the backchannel vocabulary is not even listed, which makes the annotation rule incomplete. Every F1 gain in the paper is computed on these labels, so the 10%/33% improvement may describe detection of ASR segment boundaries plus a word list rather than the conversational phenomena named in the title.\n\nAlso, all results are point estimates with no error bars, and the comparison is against reimplemented baselines on the authors' own benchmark. That is common in this subfield, but it should be phrased as best-on-MM-F2F, not unqualified SOTA. The 'first tri-modal' claim also needs to be squared with Kurata et al. 2023.\n\nThe dataset can carry the paper if the authors fix the label question: publish the backchannel vocabulary, run a human validation that actually labels a sample of TURN/BC events, report variance, and qualify the proxy nature. If the proxies track human judgments, the paper is solid; if not, it is still a well-defined benchmark for an automatic task, just not for turn-taking and backchannel as natural behaviors.\n\nRecommendation: send to peer review. The dataset will be used, and reviewers should push for label validation rather than desk-reject.","headline":"A genuinely useful tri-modal dataset and a flexible fusion baseline, but the headline numbers rest on automatic proxy labels that the manual validation never actually checks.","tokens_in":15804,"tokens_out":2914,"would_cite":true,"duration_ms":29977,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that fusing text, audio, and face video predicts turn-taking and backchannel moments better than earlier baselines on a new 210-hour face-to-face conversation dataset.","keywords":["turn-taking prediction","backchannel prediction","multimodal fusion","human-machine conversation","face-to-face dataset","full-duplex dialogue","low-rank fusion","active speaker detection"],"falsifier":"Re-annotate a random sample of MM-F2F clips with trained human raters who mark the moment a turn becomes complete and every suitable backchannel, then evaluate the same model on those human labels; if agreement between the pipeline labels and human labels is low, or if the model's F1 on the human-labeled subset drops far below the reported 0.811 and 0.906, the claimed improvements would be specific to the proxy task rather than to natural conversational timing.","tokens_in":14772,"feed_emoji":"🗣️","tokens_out":8590,"duration_ms":79767,"temperature":0.7,"pith_summary":"The paper tries to show that a conversational system can predict, word by word, whether the current speaker will keep talking, a listener should take the floor, or a short acknowledgment like \"I see\" is due. To make that test possible, it builds a 210-hour face-to-face conversation dataset with word-level labels and a model that fuses text, audio, and cropped face video. On this dataset the fused model beats earlier text-only, text-audio, and tri-modal baselines, raising F1 by about 10 percent for turn-taking and 33 percent for backchannel prediction. If the result holds, spoken interfaces can time their responses like human listeners instead of waiting for an explicit end-of-speech signal.","feed_headline":"Tri-modal model lifts backchannel F1 by 33 percent","feed_subtitle":"Built on 210 hours of face-to-face conversation, the system predicts when to yield or acknowledge.","key_machinery":"The load-bearing component is a flexible low-rank multimodal fusion module. It starts from the tensor outer product of the three uni-modal features and decomposes that weight tensor into a sum of rank-one factors, one per modality, so the fused representation is an element-wise product of per-modality projected features. An indicator function substitutes a constant one for any missing modality, reducing the same fused expression to a bi-modal or uni-modal form, and random modality dropout during training makes the tri-modal model robust to missing inputs at inference. This module is what lets the framework claim that any combination of text, audio, and video works after a single training run.","core_discovery":"The paper's central claim is that linguistic, acoustic, and visual face signals are jointly sufficient to predict keep, turn-taking, and backchannel labels at word level in dyadic face-to-face conversation, and that a low-rank fusion that preserves inter-modality products while tolerating missing modalities outperforms earlier fusion designs. The authors support this with a fully automatic collection pipeline: speech recognition supplies word-level transcripts and timestamps, speaker verification and active speaker detection assign each utterance to one of the two faces, the last word of each detected utterance is labeled TURN, and a fixed vocabulary marks BACKCHANNEL. The resulting MM-F2F dataset spans about 1.5 million words, 20 million frames, 169,029 utterances, and 955 speakers. With GPT-2, HuBERT, and VideoMAE as uni-modal encoders, the tri-modal model reaches 0.823 accuracy and F1-scores of 0.806 for keep, 0.811 for turn-taking, and 0.906 for backchannel prediction, exceeding reimplemented baselines by roughly 0.1 on keep and turn and more than 0.2 on backchannel.","pith_inferences":["Because TURN and BACKCHANNEL are defined by ASR sentence boundaries and a fixed word list, the 10 percent and 33 percent gains are proven for those proxies; whether they transfer to what a human listener would call the right moment to speak or acknowledge is an open question the paper does not settle.","The backchannel vocabulary labels may overlap heavily with short common tokens, so part of the backchannel gain could come from lexical regularity rather than from genuinely multimodal timing; a test that removes the vocabulary prior would separate these.","The flexible fusion idea is not tied to the three encoders tested here; the same indicator-function low-rank product could carry additional signals such as body motion, gaze, or gesture.","A natural stress test is to deploy the audio-plus-video version of the model in a live spoken-dialogue system with no text input and measure whether users perceive the timing as appropriate rather than merely whether the proxy label is matched."],"forward_implications":["If the reported results hold, spoken dialogue systems can decide turn-taking and backchannel timing from the speaker's words, voice, and face rather than from an explicit end-of-turn signal or a voice-activity threshold.","The same trained model can be deployed with only text and audio, only audio and video, or any other pair, making the approach usable in phone calls, video calls, and voice-only devices without retraining.","Visual cues from the face carry information that text and audio alone miss, especially for backchannels, so full-duplex systems with a camera should include the face channel.","Privacy-preserving de-identification of faces, voices, and backgrounds does not substantially degrade prediction, which supports building larger socially useful conversational datasets from in-the-wild videos.","The dataset itself provides a benchmark for word-level keep, turn, and backchannel prediction in dyadic face-to-face conversation, which earlier resources lacked."],"supporting_citations":[{"why":"Provides the TurnGPT text-only baseline and the vocabulary-matching approach adopted for backchannel annotation.","marker":"Ekstedt and Skantze, 2020"},{"why":"Supplies the text-plus-audio baseline that fuses acoustic and language-model features for turn-taking and backchannel prediction.","marker":"Wang et al., 2024a"},{"why":"Provides the tri-modal visual-cue baseline that the paper extends and compares against on MM-F2F.","marker":"Kurata et al., 2023"},{"why":"Gives the low-rank decomposition of the fusion tensor that the flexible fusion module builds on.","marker":"Liu et al., 2018"},{"why":"Defines the tensor outer-product fusion representation the paper starts from.","marker":"Zadeh et al., 2017"},{"why":"WhisperX performs the speech recognition that supplies the word and sentence timestamps used as TURN labels.","marker":"Bain et al., 2023"},{"why":"TalkNet active speaker detection links each transcribed clip to the visible speaker's face.","marker":"Tao et al., 2021"},{"why":"GPT-2 is the linguistic backbone that performed best among the text encoders tested.","marker":"Radford et al., 2019"},{"why":"HuBERT is the acoustic backbone that performed best among the audio encoders tested.","marker":"Hsu et al., 2021"},{"why":"VideoMAE is the visual backbone that encodes the last several face frames before each prediction point.","marker":"Tong et al., 2022"}],"fun_headline_variants":["Tri-modal fusion boosts backchannel F1 by 33%","New MM-F2F dataset and tri-modal model raise backchannel F1 by a third","Multimodal predictor improves backchannel F1 by 33% on face-to-face talk","Backchannel F1 up 33% with tri-modal conversation model"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the automatic annotations match real conversational moments: TURN is the last word of each speech-recognition sentence and BACKCHANNEL is any word in a fixed list, so the reported gains are for predicting those proxies.","fun_headline_variants_meta":{"raw":{"variants":["Tri-modal fusion boosts backchannel F1 by 33%","New MM-F2F dataset and tri-modal model raise backchannel F1 by a third","Multimodal predictor improves backchannel F1 by 33% on face-to-face talk","Backchannel F1 up 33% with tri-modal conversation model"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000585,"raw_usage":{"total_tokens":2767,"prompt_tokens":982,"completion_tokens":1785,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":598,"completion_tokens_details":{"reasoning_tokens":1701}},"tokens_in":598,"tokens_out":1785,"duration_ms":13741,"temperature":1.0,"reasoning_tokens":1701,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:28:37.750407+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate a random sample of MM-F2F clips with trained human raters who mark the moment a turn becomes complete and every suitable backchannel, then evaluate the same model on those human labels; if agreement between the pipeline labels and human labels is low, or if the model's F1 on the human-labeled subset drops far below the reported 0.811 and 0.906, the claimed improvements would be specific to the proxy task rather than to natural conversational timing.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the tri-modal visual-cue baseline that the paper extends and compares against on MM-F2F."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"TalkNet active speaker detection links each transcribed clip to the visible speaker's face."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"VideoMAE is the visual backbone that encodes the last several face frames before each prediction point."}],"review_version":1}