{"id":"c1e24555-37a1-4f94-838d-9c09acc1e79e","arxiv_id":"2506.21191","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A transformer-based turn-taking predictor that can be steered by textual prompts such as 'faster' or 'calmer', validated on 950+ hours of dialogue.","lead":"This paper adds a text-prompt control knob to a neural model that predicts when speakers will take turns, so a dialogue system can be told to respond faster or calmer. The authors test it on over 950 hours of Japanese conversation, using an LLM to invent prompt labels because real prompt data don't exist.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported accuracy gains likely stem from label leakage: prompts are synthesized from the same 20-second window whose future voice activity is the prediction target, so the baseline comparison in Table 1 is circular.","rationale":"The reader's weakest assumption identifies synthetic prompts as an imperfect proxy for real user instructions and notes a 'circularity burden,' which overlaps with my concern. However, I locate the load-bearing problem more precisely: the prompts are generated from the same audio window whose future voice activity is the prediction target, making the quantitative comparison in Table 1 vulnerable to label leakage rather than merely to distribution shift. This is the strongest threat to the paper's central claim because the accuracy improvement is one of the two pillars of the contribution, and the other pillar (qualitative controllability) rests on only two examples without a metric. I still do not move the verdict: conditional acceptance is appropriate because the architecture is clearly described, the prompt-reconstruction loss is a sensible auxiliary objective, and the proposed leakage-control experiment can settle the question. The condition for acceptance should be sharpened to require this ablation (and ideally real user prompts with statistical testing), which is consistent with the reader's call for stronger evaluation.","tokens_in":6028,"tokens_out":5896,"duration_ms":73193,"concrete_test":"Re-train and re-evaluate the proposed model with prompts generated only from the preceding 20-second window (or from utterance timing strictly prior to each prediction frame), so the prompt carries no information about the future bins being predicted, keeping all other settings identical. If the VAP loss and S/H balanced accuracy advantage over the original VAP disappears (or becomes statistically indistinguishable), the reported gains are attributable to leakage; if the advantage persists, the prompt-conditioning mechanism has predictive value beyond the leaked timing summary.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline quantitative result (VAP loss 2.346 vs 2.431; S/H balanced accuracy 79.80 vs 77.17) does not establish that prompt conditioning improves turn-taking prediction, because Section 3.2 generates each prompt from the utterance timing of the same 20-second segment that supplies the VAP prediction targets. A prompt such as 'speaks calmly, with deliberate pauses, before and after utterances' is a compressed text description of the ground-truth future voice activity that the model is asked to predict. The proposed model receives this summary at every frame while the original VAP baseline receives only audio, so the measured gain can be explained entirely by target leakage. The qualitative prompt-swap examples in Figures 2-3 show output changes under prompt changes, but with two examples and no quantitative controllability metric they do not validate transfer to real user instructions. The limitations paragraph acknowledges synthetic prompts and generalizability, but it does not acknowledge the within-window leakage that compromises the central accuracy comparison.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper extends the Voice Activity Projection (VAP) turn-taking model with textual prompt conditioning. Prompt embeddings are produced by a Japanese text encoder, linearly projected, and concatenated at two points in the transformer stack (post-audio-encoder and post-self-attention), with an auxiliary loss that reconstructs the prompt embedding. Because no natural prompt annotations exist in the dialogue corpora, the authors generate synthetic prompts with GPT-4.1 from voice activity timings of the same 20-second audio segments used as model input. Experiments on approximately 950 hours of Japanese dialogues compare the proposed model with the original VAP and report lower VAP loss (2.346 vs. 2.431) and higher turn-shift/hold balanced accuracy (79.80% vs. 77.17%). Two qualitative examples show that swapping user and system prompts changes predicted turn-taking timing.","tokens_in":6227,"tokens_out":3318,"duration_ms":42082,"significance":"If the reported effects are real, prompt-controllable turn-taking would be a practically useful capability for spoken dialogue systems, and the proposed architecture is a simple, clearly described extension of a well-known baseline. The use of over 950 hours of diverse Japanese dialogue data and the explicit treatment of ethical considerations are strengths. However, the central quantitative comparison is compromised because the synthetic prompt is generated from the same audio window whose future voice activity is the prediction target; the accuracy gain over the original VAP can therefore be explained by information leakage rather than by prompt conditioning. The controllability claim rests on two qualitative examples without a quantitative metric. These issues make the current evidence insufficient, but they are addressable with additional experiments.","major_comments":[{"comment":"The prompt-generation procedure creates a leakage path that invalidates the headline accuracy comparison. Section 3.2 states that GPT-4.1 is given the speaker identities and utterance start/end times for each 20-second audio segment aligned with the VAP model's training input, and that the prompt is generated from a description of that same segment's turn-taking style. The VAP prediction target is precisely the future voice activity derived from those utterance timings. Thus the proposed model receives, at every frame, a textual summary of the ground-truth information it is asked to predict, whereas the original VAP baseline receives only the audio. The improvement in Table 1 (VAP loss 2.346 vs. 2.431; S/H balanced accuracy 79.80 vs. 77.17) is therefore not evidence that textual prompts improve turn-taking prediction; it can be fully explained by target leakage. To support the claim, the authors should generate prompts from a preceding or held-out window, or otherwise ensure the prompt contains no information about the prediction target, and should compare against a baseline that receives the same auxiliary timing information in a non-textual form.","section":"Section 3.2, Table 1"},{"comment":"The evaluation uses a single random 8:1:1 session-level split with no error bars, no significance testing, and no multiple-seed analysis. Given that the reported differences are small (1.6 percentage points in balanced accuracy and 0.085 in VAP loss), the reader cannot determine whether these differences are stable or within run-to-run variability. The authors should report mean and standard deviation over several splits or training runs, and ideally a paired significance test, before claiming that the proposed model improves prediction accuracy.","section":"Section 4, Table 1"},{"comment":"The qualitative demonstration of prompt controllability rests on exactly two examples with hand-prepared prompts, and no quantitative controllability metric is reported. The claim that the model 'effectively varied turn-taking timing behaviors according to the textual prompts' requires a more systematic evaluation, for example measuring turn-shift rate or response latency under systematically varied prompts across many test segments, and showing that the effect is statistically distinguishable from prompt-independent variation.","section":"Section 4, Figures 2 and 3"},{"comment":"The training loss in Eq. (1) sums L_vap, L_vad, and L_prompt with no stated weights. If the three terms are simply summed with equal weight, this should be stated; if weights were tuned, the values and selection procedure should be reported. More importantly, because the prompt embedding is concatenated at every frame and a reconstruction loss encourages the model to retain it, the model may learn to rely on the prompt rather than on the audio; the current experiment cannot separate these contributions because the prompt is generated from the audio's own future activity.","section":"Section 2.2, Eq. (1)"}],"minor_comments":[{"comment":"The text says prompts are provided 'for each participant independently,' but Figure 1 shows a single 'Prompt for participant 1' path and mentions that the same processing is applied to participant 2; please clarify whether each participant has a distinct prompt and how the two prompts are encoded and combined when they differ.","section":"Section 2.2"},{"comment":"The statement that 'Model configurations and training parameters were consistent with those described in the original work' is not sufficient for reproducibility because the original VAP paper does not define the prompt-related hyperparameters introduced here, such as the prompt embedding projection dimension, the learning rate, batch size, number of epochs, or the loss weighting; please provide a complete hyperparameter table.","section":"Section 4"},{"comment":"The example utterance timing contains overlapping intervals (e.g., B 13.813-16.725 and A 15.123-15.898), which is surprising for a VAD-based annotation of a dyadic conversation; please clarify whether this is intentional overlapped speech, a formatting artifact, or an error in the example.","section":"Appendix A"},{"comment":"The description says VAP logits represent a probability distribution over 256 discrete states corresponding to four future time bins, but the aggregation into p_now and p_future is not formally defined; please specify how the 256 states map to the 0-600 ms and 600-2000 ms intervals.","section":"Section 2.1"},{"comment":"The Limitations section acknowledges the reliance on synthetic prompts and the restriction to Japanese data, but it does not acknowledge the more serious issue that the prompts are generated from the same audio segment whose future voice activity is the prediction target; this should be discussed explicitly and addressed experimentally.","section":"Limitations"}],"recommendation":"major_revision","confidential_remarks":"The core idea is interesting and the architecture is sensible, but the main quantitative result is not currently interpretable because of the leakage path in Section 3.2. The paper would need either a re-designed evaluation (prompts from non-overlapping windows, an equivalent-information baseline, and significance testing) or a reframing that does not claim accuracy improvement over the original VAP. I believe this is fixable within the scope of a revision, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The idea here is real and worth testing: prompt-conditioned turn-taking prediction. The authors are the first to integrate textual prompt embeddings into the VAP transformer stack, and the architecture is clean—prompt embeddings are concatenated at two points, plus a reconstruction loss. The qualitative examples in Figures 2 and 3 show that swapping prompts changes predicted turn timing in the intended direction, which is the paper's most convincing evidence.\n\nBut the headline accuracy comparison is not trustworthy. Prompts are generated by GPT-4.1 from the same 20-second audio window that supplies the VAP prediction targets. So at every frame, the proposed model receives a text summary that describes the turn-taking style of the entire window, including the future voice activity it is being scored on. The original VAP baseline gets only audio. The reported VAP loss gain (2.346 vs 2.431) and balanced accuracy gain (79.80 vs 77.17) can be fully explained by this leakage. The limitations section acknowledges synthetic prompts but misses this within-window leakage, and that is a real gap.\n\nSmaller issues: there is a single train/validation/test split with no error bars or significance tests, so the numeric improvements are not established even without leakage. The proposed model also adds an auxiliary prompt reconstruction loss, so the comparison to the original VAP conflates prompt information with an extra training objective. The controllability evidence rests on two qualitative examples; no quantitative controllability metric is reported.\n\nThis paper is for dialogue systems researchers interested in controllable turn-taking. It deserves a serious referee, but not acceptance in its current form. The accuracy claim should be reframed or removed, and the controllability claim needs a real evaluation: predefined prompts, held-out prompt styles, a controllability metric, and a baseline with a non-informative prompt summary to separate leakage from genuine conditioning.","headline":"A genuinely new prompt-conditioning architecture for turn-taking prediction, but the headline accuracy gain is compromised by within-window label leakage and needs a corrected evaluation.","tokens_in":6738,"tokens_out":2908,"would_cite":false,"duration_ms":35248,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Textual prompts like “faster” or “calmer” can steer a turn-taking predictor’s timing, and adding them also improves prediction accuracy.","keywords":["turn-taking prediction","voice activity projection","textual prompts","spoken dialogue systems","conversational robots","prompt-conditioned model","LLM synthetic prompts","VAP"],"falsifier":"Take a fixed real-world instruction such as “respond faster and take the lead” and feed it with test audio segments whose actual style varies; if the predicted turn times do not move systematically toward faster responses, or if reassigning each audio segment to a randomly chosen prompt leaves VAP loss and balanced accuracy unchanged, then the reported gains come from prompt-segment leakage rather than genuine prompt understanding.","tokens_in":5828,"feed_emoji":"🗣️","tokens_out":6227,"duration_ms":64092,"temperature":0.7,"pith_summary":"This paper establishes that turn-taking prediction in spoken dialogue systems can be controlled by natural-language prompts instead of fixed model behavior. It extends the voice activity projection (VAP) transformer so that a short instruction for each participant, embedded as text, is injected into both the per-channel self-attention and the cross-channel attention stages. On roughly 950 hours of Japanese human-human dialogue, with prompts synthesized by an LLM from utterance timing, the prompt-conditioned model achieves lower VAP loss (2.346 vs. 2.431) and higher balanced accuracy (79.80% vs. 77.17%) than the original VAP model. Qualitative examples show that swapping prompts such as “respond immediately” and “speak slowly and deliberately” visibly shifts when the system is predicted to take its turn. The practical interest is that dialogue systems and robots could be adapted to users and contexts simply by changing an instruction.","feed_headline":"Text prompts like “faster” and “calmer” steer turn-taking prediction","feed_subtitle":"Adding prompt embeddings to a voice-activity transformer lowers VAP loss and shifts predicted turn timing on command.","key_machinery":"The central object is the voice activity projection (VAP) transformer, a model that maps two channels of speech audio through a contrastive predictive coding (CPC) encoder, per-channel self-attention, and a cross-channel attention transformer to predict voice activity over the next two seconds. The action is the prompt-conditioning mechanism: a 1792-dimensional sentence embedding of each participant's instruction is linearly projected and concatenated with the audio features at two depths, and a reconstruction head enforces that the prompt signal survives to the cross-attention output. That dual injection plus prompt reconstruction is what lets the same underlying predictor change its turn-taking timing when the text changes.","core_discovery":"The central claim is that textual prompts can be turned into a first-class conditioning signal for turn-taking prediction. The authors build on the VAP architecture, which predicts future voice activity from a stereo audio stream, and concatenate prompt embeddings into the feature vectors at two points: right after the audio encoder and again after the per-channel self-attention transformer, before cross-channel attention. An auxiliary loss reconstructs the prompt embedding from the cross-attention output, so prompt information is pushed through the whole network. Trained with synthetic prompts generated by GPT-4.1 from the utterance timing in each audio segment, the model improves over the baseline on both VAP loss and balanced accuracy, and the paper demonstrates in worked examples that the predicted timing of system turns moves in the direction requested by the prompt. To the authors' knowledge, this is the first system to explicitly direct turn-taking prediction with textual prompts.","pith_inferences":["Beyond the paper: synthetic prompts are generated from the same 20-second audio window they are applied to, so the prompt may leak information about that segment's timing that a real user issuing an instruction in advance would not have; a fixed instruction like “respond faster” might not shift behavior as cleanly as the examples suggest.","Beyond the paper: a natural next step is to test transfer with held-out prompt styles or with no prompt at inference time, to see whether the model falls back to baseline behavior or distorts predictions when the prompt is absent.","Beyond the paper: the same conditioning mechanism could plausibly be ported to multilingual or backchannel-prediction variants of VAP, where different prompt languages or listener styles would be desirable, but that would require collecting natural prompt data in each language."],"forward_implications":["A dialogue system can reconfigure its own turn-taking style at runtime by swapping the prompt for the system channel, without retraining or changing the audio encoder.","Because prompts are conditioned per participant, the system can hold different behavioral specifications for the user and for itself in the same conversation.","If the accuracy gain is real, prompt text carries information about upcoming speech activity that the audio encoder alone does not extract, making prompts a useful auxiliary input rather than just a control knob.","The training recipe—synthesize prompts from timing statistics, then train with a reconstruction loss—is reusable for any turn-taking dataset that lacks natural prompt annotations."],"supporting_citations":[{"why":"Provides the base VAP architecture and the training configuration that the proposed model extends with prompt embeddings.","marker":"Ekstedt and Skantze, 2022"},{"why":"Supplies the contrastive predictive coding (CPC) audio encoder that turns each participant's waveform into features.","marker":"Riviere et al., 2020"},{"why":"The prompt-guided text-to-speech work that motivates concatenating prompt embeddings into the feature stream.","marker":"Guo et al., 2023"},{"why":"Defines the balanced-accuracy evaluation metric for turn shift versus hold prediction and the multilingual VAP extension.","marker":"Inoue et al., 2024"},{"why":"Contributes the travel agency task dialogue corpus used in the 953.5-hour Japanese training mixture.","marker":"Inaba et al., 2022"},{"why":"Contributes the ERICA human-robot Wizard-of-Oz dialogue data used alongside the human-human corpora.","marker":"Inoue et al., 2025a"}],"fun_headline_variants":["Text prompts like 'faster' or 'calmer' now steer turn-taking","First model to control turn-taking with text prompts","Prompt-guided turn-taking: say 'faster' and it obeys","Turn-taking prediction now responds to textual instructions","Using LLM prompts to adjust dialogue timing prediction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that LLM-written prompts derived from each audio segment's speaking times behave like real instructions a user would give, so the measured controllability and accuracy gain are not artifacts of the prompt leaking information about the segment that the audio encoder cannot see.","fun_headline_variants_meta":{"raw":{"variants":["Text prompts like 'faster' or 'calmer' now steer turn-taking","First model to control turn-taking with text prompts","Prompt-guided turn-taking: say 'faster' and it obeys","Turn-taking prediction now responds to textual instructions","Using LLM prompts to adjust dialogue timing prediction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000195,"raw_usage":{"total_tokens":1321,"prompt_tokens":874,"completion_tokens":447,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":490,"completion_tokens_details":{"reasoning_tokens":365}},"tokens_in":490,"tokens_out":447,"duration_ms":5243,"temperature":1.0,"reasoning_tokens":365,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:32:00.425646+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fixed real-world instruction such as “respond faster and take the lead” and feed it with test audio segments whose actual style varies; if the predicted turn times do not move systematically toward faster responses, or if reassigning each audio segment to a randomly chosen prompt leaves VAP loss and balanced accuracy unchanged, then the reported gains come from prompt-segment leakage rather than genuine prompt understanding.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the base VAP architecture and the training configuration that the proposed model extends with prompt embeddings."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the contrastive predictive coding (CPC) audio encoder that turns each participant's waveform into features."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The prompt-guided text-to-speech work that motivates concatenating prompt embeddings into the feature stream."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the balanced-accuracy evaluation metric for turn shift versus hold prediction and the multilingual VAP extension."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes the travel agency task dialogue corpus used in the 953.5-hour Japanese training mixture."}],"review_version":1}