{"id":"73a69c8c-3452-45e1-b6eb-0f749533404f","arxiv_id":"2507.23298","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A VAP-based model predicts the timing and type (short, long, long_p) of listener nodding in real time, and multi-task learning with backchannel prediction plus pretraining improves type prediction.","lead":"This paper presents a real-time model that predicts when and what type of nod an attentive listening avatar should make, using audio from both the speaker and the listener. In a user study, the proposed method was rated significantly more natural, attentive, and encouraging than a conventional system that always nods together with a verbal backchannel.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 203-dialogue VAP pretraining set appears to include the 9 test dialogues (Section 4 excludes nothing), so reported F1 and significance may reflect test-set leakage rather than clean generalization.","rationale":"The most load-bearing issue is not the re-enactment per se but the apparent composition of the pretraining corpus. The paper never states that the 9 test dialogues are excluded from the 203-dialogue pretraining set; the natural reading of Section 4 is that the whole ERICA WOZ dataset, including all 90 attentive-listening dialogues, is used for VAP pretraining. Since the nodding head is then finetuned on the same test recordings, the objective F1 scores and significance tests are not clean estimates. This is a testable, unambiguous check: inspect the released training scripts or ask the authors for the pretraining split. If the test set is included, a rerun is mandatory. The reader's concern about single-operator re-enacted labels is legitimate and also threatens ecological validity, but even if the labels were perfect, the leakage issue would still corrupt the reported numbers; conversely, fixing the leakage does not fix the label validity issue. Thus I partially agree with the reader. Because the manuscript can be repaired by clarifying/restricting the pretraining set and rerunning, the appropriate verdict remains CONDITIONAL rather than REJECT; if the authors cannot provide the split or the rerun fails, the objective claims should be withdrawn.","tokens_in":10691,"tokens_out":14294,"duration_ms":172153,"concrete_test":"Inspect the released code and data splits (or ask the authors) and check whether any of the 9 test dialogue IDs appear in the list of 203 dialogues used for VAP pretraining. If they do, rerun the MT w/BC,PT condition with pretraining on the 72 training dialogues plus the 131 non-attentive-listening dialogues only, and recompute Tables 4-8. If the equally-averaged F1 falls below the non-pretrained MT w/BC model, or the bootstrap 95% confidence interval excludes the reported 30.04, the headline quantitative claim is not supported by clean generalization.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4 states that VAP pretraining uses 203 Wizard-of-Oz dialogues collected with ERICA, 'including 72 dialogues used as data for training described in Section 2.' The attentive-listening corpus in Section 2 is 90 dialogues split 72/9/9 for training/validation/test, and the remaining 113 of the 203 are job-interview and first-meeting dialogues. As written, the pretraining set is a superset of the entire attentive-listening corpus, and no sentence excludes the 9 validation or 9 test dialogues. If the test audio appears in pretraining, the self-attention and cross-attention layers are trained on the exact test recordings before the nodding head is finetuned. Reported F1 gains in Tables 2, 4-7 and the bootstrap significance tests in Tables 3 and 8 would then be inflated by transductive leakage, not clean generalization. This concern is more decisive than the single-operator re-enactment issue: even if the re-enacted gestures were perfectly natural, the measured improvement could be an artifact of pretraining/test overlap. The paper must state the exact composition of the 203-dialogue pretraining set and, if it includes held-out dialogues, rerun the experiments with pretraining restricted to training/validation data only.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a VAP-based model for real-time prediction of listener nodding, predicting both the timing and the type of nod (short, long, long_p) from the audio of both dialogue participants. The model extends the VAP architecture with an extra linear head for nodding, adds multi-task learning with verbal backchannel prediction, and uses pretraining on a larger Wizard-of-Oz dialogue corpus. Frame-level F1 scores are reported for timing (best 55.93) and for the three types (roughly 22–39), with bootstrap significance tests; the authors show that reducing the processing rate to 10 Hz achieves real-time operation on CPU with little F1 loss. The model is integrated into an avatar attentive listening system, and a crowdsourced subjective study (n=45) finds that separately predicting backchannels and multiple nodding types yields significantly higher ratings in naturalness, attentiveness, and facilitation than a conventional system that always couples nods with predicted verbal backchannels.","tokens_in":10996,"tokens_out":4892,"duration_ms":54775,"significance":"If the results are clean, this is a useful and reasonably novel contribution: it extends the VAP framework from turn-taking and backchannel prediction to continuous, real-time, multi-class nodding prediction, and it provides an end-to-end avatar integration with a subjective evaluation. The public release of code and trained models is a concrete reproducibility strength. However, the quantitative claims are tempered by two concerns: the pretraining corpus may overlap the test dialogues (which would invalidate the reported generalization numbers), and the ground-truth nods were re-enacted after the fact by a single operator. The absolute F1 scores are also modest, and most comparisons lack confidence intervals or multiple-comparison control, so the strength of the evidence is weaker than the text sometimes suggests.","major_comments":[{"comment":"The composition of the 203-dialogue VAP pretraining set is not stated precisely enough. The paper says it 'includes 72 dialogues used as data for training described in Section 2,' but Section 2 divides the attentive-listening corpus into 72 training, 9 validation, and 9 test dialogues; the text does not state that the 9 validation and 9 test dialogues are excluded from pretraining. Under a strict reading, the pretraining set is a superset of the entire attentive-listening corpus, so the self-attention and cross-attention layers would be trained on the exact test recordings before the nodding head is fine-tuned. In that case the F1 scores in Tables 2, 4–7 and the bootstrap significance results in Tables 3 and 8 would reflect transductive leakage rather than clean generalization. The authors must state the exact overlap between the 203 pretraining dialogues and the 90 attentive-listening dialogues, and if any held-out dialogue appears in pretraining, they must rerun the experiments with pretraining restricted to the 72 training dialogues.","section":"Section 4 (Experimental Evaluation)"},{"comment":"The ground-truth nodding labels are produced by a single operator re-enacting listener gestures after the fact while reviewing recorded dialogues. No validation is provided that the re-enacted timing, duration, and movement-range categories (short, long, long_p) match naturally occurring attentive-listening nods in live interaction. Because every objective F1 number, the model's training signal, and the nodding motions used in the subjective evaluation are derived from these re-enacted labels, the central quantitative claims are contingent on the fidelity of this re-enactment. The paper should provide at least an inter-operator or repeated-recording consistency check, or explicitly scope the claims to the operational setting of this re-enactment procedure.","section":"Section 2.1 (Recording of Listener Gestures)"},{"comment":"The text states that VAP pretraining was 'confirmed' to improve performance in the timing-and-type task, but the bootstrap t-tests for pretrained versus non-pretrained models are not statistically significant (ST w/ PT vs. ST, p=0.138; MT w/ BC, PT vs. MT w/ BC, p=0.111). The same is true in the timing task (Table 3: p=0.130 and p=0.454). The claims in Sections 4.1 and 4.2 should be revised to separate descriptive F1 gains from statistically supported ones, and the conclusion that 'VAP pretraining with general dialogue data improves performance' should be softened accordingly.","section":"Section 4.2 and Table 8"}],"minor_comments":[{"comment":"The 'self-feedback' of predicted backchannel results to the listener's speech signal input is mentioned but never specified; please clarify how this feedback is implemented (e.g., as an additional input feature, a gating mechanism, or a loss-level interaction).","section":"Section 3.2"},{"comment":"The claim that reducing the frame rate to 10 Hz causes 'no substantial decline' in F1 is based on point estimates only; no confidence intervals or statistical tests are provided for the differences across input lengths and frame rates.","section":"Section 4.3, Tables 9 and 10"},{"comment":"The Random baseline is described only as 'always predicting nodding in all frames' for the timing task. Please clarify what the Random baseline does in the type-prediction task (e.g., always predicts all four classes with uniform probability, or cycles through classes).","section":"Section 4, Tables 2 and 4-7"},{"comment":"Please clarify how the nine audio recordings were assigned to the three worker sets and whether each recording came from a distinct user or session; this affects the interpretation of the per-method averaging across the 45 participants.","section":"Section 5.1"},{"comment":"There are several small typographical issues, including 'lonp_p nodding' and 'predicton' in Section 4.2, and inconsistent spacing in Table 2; these should be corrected in the camera-ready version.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The pretraining/test overlap question is the most important risk and should be resolved before acceptance. If the 9 validation and 9 test dialogues are indeed part of the 203-dialogue pretraining set, the objective evaluation would need to be redone; if they are not, a one-sentence clarification would suffice. The paper also leans heavily on the authors' own VAP and backchannel work, but the central nodding claim is independently compared against baselines, so I do not see this as a circularity problem."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, here's my read on 2507.23298.\n\nThe genuinely new thing here is using VAP-style future voice-activity prediction to predict not just timing but three explicit nodding types (short/long/long_p) in real time, with multi-task learning on backchannel prediction. That's a real extension over prior prosody-based or backchannel-conditioned nod models. The architecture is straightforward, and they ship code and trained models, which I appreciate. The real-time CPU analysis is also useful and honestly presented.\n\nThe experiments are extensive: timing and type F1, statistical tests on the comparisons, and a user study with 45 workers. The user study is interesting: separating backchannel and nod generation into independent predictions actually beat always-coupling them on naturalness, attentiveness, and facilitation. That's a nice result, if it holds.\n\nBut I have two concerns, and one is potentially decisive.\n\nThe first is the VAP pretraining set. Section 4 says they pretrain on 203 WOZ dialogues 'including 72 dialogues used as data for training described in Section 2.' The attentive-listening corpus has 90 dialogues; 72 are train, 9 validation, 9 test. As written, the 203 could be a superset of all 90, meaning the test audio was seen during pretraining. That would make the F1 gains and the significance tests transductive, not clean generalization. The paper never says the 9 validation/test dialogues are excluded from the 203. This needs to be stated explicitly, and if they are included, the experiments need rerunning with pretraining restricted to the training split. This is more worrying than the re-enactment issue below.\n\nSecond, the nodding ground truth is a single operator re-enacting listener gestures after the fact. That is a real limitation, especially because the same operator is the one who controlled ERICA and knows the dialogue. It's not fatal if the re-enactment is faithful, but it's a major validity caveat.\n\nSmaller issues: absolute F1 scores are modest (55.9 timing, 22-39 types), there are no confidence intervals and multiple t-tests without correction. The subjective evaluation also uses audio that was originally recorded while the user was interacting with the proposed method, which could favor it. The user study itself is otherwise okay.\n\nBottom line: this is a plausible, competent application that deserves serious referee time, but the pretraining overlap question must be resolved before the reported numbers are citable. If the authors clarify and rerun if needed, this becomes a solid contribution. I would not cite the numerical results as-is, but I'd keep an eye on the revision.","headline":"Novel VAP extension for real-time nod-type prediction, but the pretraining set may include the test dialogues—that ambiguity needs to be settled before the numbers can be trusted.","tokens_in":11512,"tokens_out":4249,"would_cite":false,"duration_ms":45307,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that an audio-only extension of VAP can predict both the timing and type of listener nodding in real time, making an avatar feel more natural, attentive, and encouraging than conventional backchannel-coupled nodding.","keywords":["Nodding Prediction","Voice Activity Projection","Backchannel Prediction","Multi-task Learning","Attentive Listening System","Avatar Nonverbal Behavior","Real-time Generation","Spoken Dialogue Systems"],"falsifier":"Record real dyadic attentive-listening interactions with motion capture of the listener's head, train the same audio-only VAP-based predictor on those naturally occurring nods, and run the avatar against the conventional 'nod with every backchannel' system; the central claim fails if the model no longer beats that baseline on attentiveness or if its timing F1 drops to near chance.","tokens_in":10503,"feed_emoji":"🤖","tokens_out":8292,"duration_ms":84695,"temperature":0.7,"pith_summary":"An attentive-listening avatar only feels attentive if it reacts while the user is speaking, and a single generic nod repeated on every backchannel quickly feels mechanical. This paper tries to establish that an audio-only model can do better: it can predict, about half a second before a nod would be transcribed, not just that a listener will nod but whether the nod will be short, long, or a long nod with an upward swing, and can run in real time on a laptop CPU. The authors build this by extending the voice activity projection (VAP) model, which already anticipates turn-taking from both speakers' audio, with an extra head that classifies nodding frames, and by co-training that head with verbal backchannel prediction and pretraining on general dialogue data. In crowdsourced ratings, an avatar driven by the model is judged significantly more natural, attentive, and encouraging than the conventional system that always performs a single nod type together with each backchannel. If the result holds, spoken dialogue agents can produce human-like listening behavior using only microphones, with no camera or body-tracking input.","feed_headline":"Real-time audio predicts three nod types for attentive avatars","feed_subtitle":"VAP-based model with backchannel multi-task learning beats always-coupling nods to verbal backchannels in user ratings.","key_machinery":"The carrying mechanism is the voice activity projection (VAP) architecture: each participant's audio is encoded with contrastive predictive coding into a representation of the current and immediately upcoming speech activity, then passed through self-attention and cross-attention transformers so the model can relate the two channels. The paper adds a single linear layer that classifies each processed frame into one of four classes (no nod, short, long, long_p), and optionally a second linear layer that predicts verbal backchannel timing; the total loss combines the nodding cross-entropy with the original VAD/VAP losses and a weighted backchannel loss. Two design choices make the head usable for an interactive avatar: the ground-truth nod intervals are shifted 500 ms earlier during training so the model predicts a nod before it is physically displayed, and the processing rate can be dropped to 10 Hz, which keeps the real-time factor below 1.0 on a CPU. The three types are defined by movement range and upward swing: short nods have small range, long nods have large range without swinging up, and long_p nods combine large range with an upward swing.","core_discovery":"On the paper's own terms, the central discovery is that listener nodding, which is nonverbal and often assumed to need visual or kinematic input, can be projected from speech alone in a continuous, frame-wise manner. Extending VAP with a further linear layer yields an F1 of 55.93 percent on the binary timing task and per-type F1 scores of 28.86, 39.17, and 22.09 percent for short, long, and long_p nods in the four-way timing-and-type task. Multi-task learning with backchannel prediction gives a statistically significant gain in the timing-and-type task (p = .049 and .012 for the two MT-versus-ST comparisons), and VAP pretraining on general dialogue data adds further improvement. Reducing the processing frame rate from 50 Hz to 10 Hz keeps the real-time factor below 1.0 on a CPU while causing no substantial drop in F1, so the model can be embedded in a deployed avatar. In the subjective evaluation, separating backchannel and nod generation is rated significantly higher than always nodding together with backchannels on naturalness (p = .019), attentiveness (p < .001), and facilitation (p = .012).","pith_inferences":["If the audio-only projection generalizes, the same VAP backbone could be extended to eye gaze, head motion, and facial-expression timing, since the pretrained speech-activity representation is not tied to nodding.","The 500 ms look-ahead makes the nod prediction a candidate turn-taking feature: an early long_p nod may mark an upcoming floor shift, which could let the system decide whether to yield or hold the turn.","Because the multi-task gains concentrate on long and long_p nods, a finer-grained model that conditions nod type on the exact lexical form of the co-occurring backchannel ('mm', 'uh-huh', 'right') might further improve type accuracy.","The recorded-audio evaluation tests the avatar's behavior on fixed user speech; a fully interactive study, in which the user reacts to the avatar's nods while speaking, could reveal whether the attentiveness effect grows or shrinks in a live loop."],"forward_implications":["An avatar can convey attentive listening with audio-only input, removing the need for a camera or body-tracker in deployment.","Because the model predicts roughly 500 ms ahead, the avatar's mechanical latency can be absorbed and the nod can land at the moment a human listener would nod.","Joint prediction of backchannel timing and nod type is more effective for large-movement nods (long, long_p) than for small ones, pointing to a tighter coupling between lexical responses and large gestures.","A 10 Hz processing rate is sufficient for real-time use on a laptop-class CPU without a substantial accuracy cost, so the method is deployable outside GPU-equipped servers.","Letting backchannel and nodding occur independently, rather than always together, changes users' perception of naturalness, attentiveness, and encouragement."],"supporting_citations":[{"why":"Supplies the VAP architecture and its VAD/VAP losses that the proposed model extends with a nodding head.","marker":"[19]"},{"why":"Provides the VAP-based backchannel prediction model used for multi-task learning and as the backchannel baseline in the subjective evaluation.","marker":"[20]"},{"why":"Establishes that nodding with an upward swing co-occurs more with assessment backchannel, motivating the long_p type.","marker":"[21]"},{"why":"Shows that nod movement range varies with co-occurring backchannel type, motivating the short/long distinction.","marker":"[18]"},{"why":"Demonstrates that VAP-based models can run in real time, supporting the real-time processing analysis.","marker":"[22]"},{"why":"Describes the ERICA android and the Wizard-of-Oz dataset from which the nodding training data were recorded.","marker":"[23]"},{"why":"Provides an earlier model that simultaneously predicts nodding and backchannel, which the multi-task results are compared against conceptually.","marker":"[17]"}],"fun_headline_variants":["Audio predicts nod timing and type for attentive avatars in real time","Speech-driven nodding: model predicts timing and type, beats sync","Real-time nod type prediction from audio outperforms backchannel sync","Nod timing and type from speech alone: real-time avatar system","Multi-task audio model predicts nodding, rated more natural and attentive"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The training labels come from a single operator re-enacting listener gestures while watching recorded dialogues after the fact, not from the original live interaction, so the model's success depends on that re-enactment faithfully matching the timing and form of natural attentive-listening nods; if the re-enactment is stylized or biased, the learned mapping may not transfer to real users.","fun_headline_variants_meta":{"raw":{"variants":["Audio predicts nod timing and type for attentive avatars in real time","Speech-driven nodding: model predicts timing and type, beats sync","Real-time nod type prediction from audio outperforms backchannel sync","Nod timing and type from speech alone: real-time avatar system","Multi-task audio model predicts nodding, rated more natural and attentive"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000202,"raw_usage":{"total_tokens":1403,"prompt_tokens":988,"completion_tokens":415,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":604,"completion_tokens_details":{"reasoning_tokens":327}},"tokens_in":604,"tokens_out":415,"duration_ms":4933,"temperature":1.0,"reasoning_tokens":327,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T10:51:06.919886+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Record real dyadic attentive-listening interactions with motion capture of the listener's head, train the same audio-only VAP-based predictor on those naturally occurring nods, and run the avatar against the conventional 'nod with every backchannel' system; the central claim fails if the model no longer beats that baseline on attentiveness or if its timing F1 drops to near chance.","supporting_citations":[{"cited_title":"Voice activity projection: Self-supervised learning of turn-taking events","cited_arxiv_id":null,"evidence_quote":"Supplies the VAP architecture and its VAD/VAP losses that the proposed model extends with a nodding head."},{"cited_title":"Yeah, un, oh: Continuous and real-time backchannel prediction with fine-tuning of voice activ- ity projection","cited_arxiv_id":null,"evidence_quote":"Provides the VAP-based backchannel prediction model used for multi-task learning and as the backchannel baseline in the subjective evaluation."},{"cited_title":"Co-occurrence relations between forms of re- sponse tokens and nods.Special Interest Group on Spoken Language Understanding and Dialogue Processing (SIG-SLUD), 90:140–145, 2020","cited_arxiv_id":null,"evidence_quote":"Establishes that nodding with an upward swing co-occurs more with assessment backchannel, motivating the long_p type."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows that nod movement range varies with co-occurring backchannel type, motivating the short/long distinction."},{"cited_title":"Real-time and continuous turn-taking prediction using voice activity projec- tion","cited_arxiv_id":null,"evidence_quote":"Demonstrates that VAP-based models can run in real time, supporting the real-time processing analysis."},{"cited_title":"Spoken dialogue system for a human-like conversational robot ERICA","cited_arxiv_id":null,"evidence_quote":"Describes the ERICA android and the Wizard-of-Oz dataset from which the nodding training data were recorded."},{"cited_title":"Multimodal listener response prediction for multi-party conversation.Special Interest Group on Spoken Lan- guage Understanding and Dialogue Processing (SIG-SLUD), 96:7–12, 2022","cited_arxiv_id":null,"evidence_quote":"Provides an earlier model that simultaneously predicts nodding and backchannel, which the multi-task results are compared against conceptually."}],"review_version":1}