{"id":"53926448-daa3-4d95-85d8-146d5ac584bf","arxiv_id":"2607.27756","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Cocktail-Talker uses three action tokens and GRPO to make a speech LLM decide whether to respond, keep listening, or ignore audio in noisy multi-speaker conversations.","lead":"This paper trains a speech AI, Cocktail-Talker, to choose one of three actions—respond, listen, or ignore—when a conversation mixes several speakers and background noise, and it builds a synthetic data pipeline to generate such cocktail-party dialogs. A generalist should read it because it tackles the neglected 'whether to speak at all' problem for voice assistants in realistic social scenes.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 0.93 macro F1 measures agreement with Gemini-generated action labels; unless those labels match human turn-taking judgments, the central claim of social appropriateness is unsupported.","rationale":"The reader's weakest_assumption is exactly the load-bearing concern I identify: Gemini-generated action labels are accepted as ground truth, and all training, reward, and evaluation use the same label source. This is not an internal inconsistency—the experiments are internally coherent—but it threatens external validity. The paper's stated goal is to build assistants that 'interact more naturally and selectively in complex social environments' (Introduction), and the strongest claim is about real capability. Without human validation of the action labels, the reported 0.928/0.930 macro F1 is a measure of fit to a synthetic label distribution. The paper's 'human-in-the-loop' phrasing in the abstract is potentially misleading, since the only human verification described in Section III is for Freesound background recordings, not dialog action labels. The concrete test I propose would settle whether this concern lands: a human annotation study comparing model predictions and Gemini labels against human judgments. If the model achieves similar accuracy against human labels, the concern is resolved; if not, the conditional acceptance should be maintained or strengthened. I see no other concern as more load-bearing: missing error bars are secondary, and the baselines are matched as well as the task allows. Thus I agree with the reader's verdict and recommend no change.","tokens_in":12676,"tokens_out":2305,"duration_ms":27143,"concrete_test":"Run a human-judgment study on a stratified sample of 300–500 turns from the seen and unseen test sets (stratify by action, environment, SNR, and speaker count). Recruit at least three human annotators per turn; present each turn with the same text metadata and audio context, and ask for respond/listen/ignore. Compute: (1) human–human agreement (e.g., Fleiss kappa); (2) Gemini-label versus human-majority agreement; and (3) Cocktail-Talker's action accuracy/macro F1 against human-majority labels versus against Gemini labels. If the model's human-label macro F1 is close to the reported 0.93 and Gemini-label agreement with humans is high, the central claim survives. If model-human F1 drops by more than ~5–10 points, or human raters systematically disagree with Gemini, the reported accuracy is partly an artifact of the synthetic label distribution and the paper's real-world claim must be soft","verdict_should_be":"UNCHANGED","load_bearing_attack":"The entire training and evaluation loop depends on Cocktail-DialogGen's Gemini 3 Pro action labels. Section III states each turn is annotated with an action label; Section IV rewards exact match to those labels; Section V evaluates against the same 'Oracle Action' labels in Table II and Fig. 4. No human validation of these labels is reported—the only 'human-in-the-loop' step in Section III is verifying Freesound background recordings, not the turn-action annotations. The reported ~0.93 macro F1 therefore measures how well Cocktail-Talker replicates the generator's notion of appropriate turn-taking, not whether its decisions match human judgments about when to speak, listen, or ignore. If Gemini systematically labels ambiguous turns differently from human raters (e.g., over-labeling 'respond' or conflating 'listen' and 'ignore'), the model's strong numbers may not transfer to real conversations. This is a validity threat to the central claim, not a claim of fabrication: the target construct 'socially appropriate turn action' is operationalized by a single LLM, with no independent ground truth.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Cocktail-Talker, a speech LLM built on Qwen2.5-Omni-7B that is fine-tuned with supervised finetuning and GRPO to emit one of three turn-action tokens (respond, listen, ignore) before generating a spoken response. Training data are produced by Cocktail-DialogGen, a pipeline in which Gemini 3 Pro composes multi-speaker dialog logs with action labels, Qwen3-TTS synthesizes speech, and a deterministic assembler mixes in background noise from Freesound/DEMAND. On the paper's synthetic test sets, Cocktail-Talker reports approximately 0.93 macro F1 for turn action accuracy, outperforming several strong speech LLMs. Ablations show the action-token vocabulary and GRPO improve the decision, and robustness analyses cover SNR, number of speakers, anonymity of names/roles, conversation vibe, and context length.","tokens_in":12943,"tokens_out":3868,"duration_ms":44210,"significance":"If the action labels are trusted, this is a useful contribution: it addresses the under-explored 'speaking side' of multi-speaker conversation for speech LLMs, introduces a scalable annotated data pipeline, and demonstrates a practical training recipe with a clear code release. The ablations are informative — GRPO mainly recovers respond recall, and the model degrades gracefully in noise and transfers to unseen environment categories. The central uncertainty is external validity: the evaluation target is the same Gemini-generated label distribution used for training, and no human validation anchors the construct of socially appropriate turn-taking.","major_comments":[{"comment":"The ground-truth action labels used for training (Section IV, reward definition) and for evaluation (Table II, Fig. 4 'Oracle Action') are all produced by the same Gemini 3 Pro dialog composer in Cocktail-DialogGen. The paper claims socially appropriate turn-taking in the Abstract and Conclusion, but no human validation of these labels is reported. The only human step in Section III is verifying Freesound background recordings, not the turn-action annotations. Please add a human rater study on a sample of turns (e.g., 300–500) that measures agreement with the Gemini labels, or evaluate on an existing multi-party conversation corpus with human-annotated addressee/turn-taking behavior. Without such evidence, the 0.93 macro F1 measures internal consistency with the data generator rather than assistant quality, which is load-bearing for the central claim.","section":"Sections III and V"},{"comment":"No confidence intervals, bootstrap estimates, or significance tests are reported. Several key comparisons are based on point estimates from a single run. For example, the difference between SFT without action tokens and SFT with action tokens is 0.903 vs 0.901 (seen) and 0.907 vs 0.903 (unseen), well within likely sampling noise. Even the GRPO gain (+2.5 pp) may be seed-dependent. Please report means and standard deviations over at least three training seeds or bootstrap CIs over test items, and state whether reported differences are statistically reliable. This is needed to support the ablation and the claimed improvement from GRPO.","section":"Section V, Tables II and III"},{"comment":"The 'unseen' environments are generated by the same Cocktail-DialogGen pipeline — same Gemini dialog composer, same Qwen3-TTS voices, same assembler and noise-mixing procedures. Thus the generalization result is limited to new environment categories within the same synthetic distribution; it does not provide evidence of transfer to real conversational audio. Please qualify the claim in Section V (Analysis, Fig. 5g) that the model 'generalizes well beyond the specific environments,' or supplement with a small real-recording evaluation if available.","section":"Section V, unseen environments"}],"minor_comments":[{"comment":"Typo: 'V oice' should be 'Voice'. Also clarify how the 100 Freesound recordings per category were 'verified by humans' — what criterion was applied?","section":"Section III, Table I"},{"comment":"The header 'penalized / conditionalSCORES' is missing a space; this makes the table harder to read. Consider separating the columns more clearly.","section":"Section V, Table III"},{"comment":"The text prompt includes a JSON snippet with a trailing comma (after 'role' in speaker_c) and an extra closing brace; ensure the example is syntactically valid as shown.","section":"Section IV, Input and Output Format"},{"comment":"The exact prompts used for the four instruction-following baselines are not given. Since the comparison depends on how well the task is expressed, include the full prompt templates in an appendix.","section":"Section V, baselines"},{"comment":"Figure 5 has many subplots sharing one legend; some labels (e.g., 'CafeterTraffic' in panel g) are truncated. Increase font sizes and fix abbreviations for readability.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"This is well-executed engineering with a clear pipeline and honest ablations. The main risk is that the evaluation construct — socially appropriate turn action — is defined entirely by the generator's own labels. If the authors provide a human annotation study and confidence intervals, the paper could become publishable; otherwise, the central claim would need to be weakened to 'agreement with the simulator's labels,' which would substantially reduce its significance. I recommend conditioning acceptance on these additions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper you'll want to check is Cocktail-Talker, which gives a speech LLM three action tokens—respond, listen, ignore—and trains it on a large simulated multi-speaker cocktail-party dataset. The core result: their model reaches about 0.93 macro F1 on turn-action prediction, well above prompt-based baselines like Qwen3-Omni (0.50) and Kimi-Audio (0.52). That's an impressive engineering outcome, and the contribution is genuine: as far as I can tell, the three-way turn-decision formulation for raw audio is new. Prior work either handles text-only response decisions or focuses on the listening side (separation, transcription, understanding). The paper also ships a data-generation pipeline, Cocktail-DialogGen, that composes 14,400 dialogs with TTS and mixes them with real noise at controlled SNRs. That pipeline alone could be useful to the community.\n\nWhat the paper does well: the ablations are clean and fairly convincing. SFT with the action tokens beats SFT without them, and GRPO adds another 2.5 points, mostly by recovering missed <|respond|> turns. The confusions make sense—listen/ignore are naturally confusable, and respond is the hard class. The robustness analysis by SNR, speaker count, anonymity, and context length is the right kind of diagnostic, and the model degrades gracefully on unseen environments. The writing is honest about limitations: fixed turn boundaries, non-streaming, no vision/spatial cues.\n\nThe soft spot is exactly what you flagged. The ground truth for 'should the assistant respond, listen, or ignore' comes from Gemini 3 Pro, which also wrote the dialogs. The GRPO reward rewards exact match to those labels, and the test set is drawn from the same generator. So the 0.93 F1 is measuring how well the model reconstructs the generator's convention for social appropriateness, not whether that convention matches human raters. That makes the headline number a bit of a closed loop. The paper does not overclaim—it says 'simulated' and never claims human validation—but readers who want real-world turn-taking will need to see human agreement on those labels or an evaluation on real conversational data. The lack of confidence intervals is minor; the bigger issue is the label anchor.\n\nOverall, this is a conditional contribution, not a breakthrough. It earns serious peer review because the task definition and the data pipeline are valuable, and the training recipe is reproducible from the paper's description. Cite it if you work on spoken dialog or speech LLMs; bring it to reading group if you're interested in how synthetic data pipelines shape evaluation.\n\nRecommendation: send it to review, but push the authors to report human agreement on a sample of the Gemini labels, or to put a small human-annotated test set next to the synthetic one. That would turn a strong system paper into a credible claim about social interaction.","headline":"Cocktail-Talker is a solid system paper with a genuinely new task formulation and a strong synthetic-data pipeline; the 0.93 accuracy is real but measures agreement with Gemini-generated labels, so treat 'socially appropriate' as an assumption rather than a demonstrated fact.","tokens_in":13438,"tokens_out":3659,"would_cite":true,"duration_ms":35887,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A speech LLM learns when to respond, listen, or ignore in noisy multi-speaker scenes.","keywords":["spoken dialog systems","multi-speaker conversation","cocktail party problem","turn-taking","action tokens","reinforcement learning","speech large language model","noisy environments"],"falsifier":"Have human annotators listen to a sample of the synthetic scenes and mark whether the assistant should respond, remain silent but attentive, or ignore; compare their labels with the model's actions. If human agreement with the model is near chance, or if retraining on human labels produces similar accuracy numbers, the current results reflect label self-consistency rather than socially appropriate turn-taking.","tokens_in":12578,"feed_emoji":"🗣️","tokens_out":5060,"duration_ms":51700,"temperature":0.7,"pith_summary":"Cocktail-Talker aims to show that a spoken-dialog assistant can do more than answer; it can decide whether to answer at all. The paper trains a speech-based large language model to emit one of three turn-action tokens — respond, listen, or ignore — before each turn, given a raw mixture of overlapping speech and noise. With supervised finetuning plus group-relative policy optimization, the model reaches about 0.93 binary respond/silent accuracy on both seen and unseen synthetic environments, outperforming existing speech LLMs that lack an explicit silent mechanism. If this holds, spoken assistants could participate selectively in real social scenes rather than assuming every utterance is directed at them.","feed_headline":"Speech AI learns when to speak, listen, or ignore in crowds","feed_subtitle":"Trained on noisy simulated dialogues, it reaches 93% accuracy on whether to answer, outperforming standard speech LLMs.","key_machinery":"The central object is a small action-token vocabulary — <|respond|>, <|listen|>, <|ignore|> — prepended to the model's output and used as targets during supervised finetuning and reinforcement learning. The reward in GRPO is the sum of an action-accuracy score (1 for a correct action, 0 otherwise) and a format-integrity score (1 if the output starts with exactly one action token and follows the structural rule), and because the action tokens are skipped when routing to the speech synthesizer, spoken quality is preserved. This token-based formulation converts an open-ended behavioral question — 'should I speak now?' — into a discrete classification the LLM can be optimized for.","core_discovery":"Cocktail-Talker is built on a speech-based large language model that receives the continuous mixed audio of all preceding turns, plus optional metadata about who is present, and must generate one of three turn-action tokens before any response. Only the <|respond|> token is followed by a synthesized spoken reply; <|listen|> and <|ignore|> both produce silence. The paper reports that supervised finetuning with these tokens gives a modest improvement over a version without them, while adding GRPO — which scores each output for action correctness and format — raises binary respond/silent accuracy to about 0.93 on both seen and unseen synthetic environments and lifts respond recall from roughly","pith_inferences":["As an editorial extension: if the LLM-generated action labels do not match human judgments about appropriate silence, the reported accuracy is an internal-consistency measure; the natural next experiment is a human listening study comparing model actions with what a person would want in the same scene.","The current system assumes fixed turn boundaries and is not streaming, so it does not yet handle barge-in or overlapping speech; adapting the action-token idea to a streaming, overlap-tolerant model is a concrete testable direction.","Because the action-token formulation is model-agnostic, the same respond/listen/ignore training recipe could be applied to other speech LLMs; the paper itself notes the idea is not limited to its base model.","The finding that anonymity hurts more than noise suggests practical systems in unknown social settings need better addressee inference — perhaps by conditioning on voice characteristics or head orientation — before turn-taking will feel natural."],"forward_implications":["A spoken assistant can make turn-taking decisions directly from raw mixed audio, without explicit speaker separation or diarization, because the model is trained on the full unprocessed mixture.","GRPO-style reinforcement learning on action accuracy is an effective way to improve turn decisions: the gain comes mainly from recovering missed responses, not from changing what is said.","Noise robustness holds across the tested SNR range (down to 0–6 dB), with only a slight accuracy drop relative to clean audio.","The largest semantic degradation occurs when names and roles are removed, indicating that inferring the addressee from context and voice is the key bottleneck for deciding whether to respond.","Transfer to unseen environments is nearly as strong as to seen ones, suggesting the learned turn-taking behavior is not tied to the specific background sounds or topics seen in training."],"fun_headline_variants":["AI picks when to answer in noisy multi-speaker chats","Speech model gains 93% accuracy on whether to reply","AI learns to stay silent when conversation isn't for it","Cocktail-Talker: AI that knows when to keep quiet","GRPO sharpens AI's turn-taking in noisy social audio"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the LLM-generated action labels used as ground truth for both training and evaluation correctly capture when a real person would want the assistant to speak, listen, or ignore; if those labels do not match human social judgment, the reported accuracy measures consistency with the data generator rather than the quality of the assistant's behavior.","fun_headline_variants_meta":{"raw":{"variants":["AI picks when to answer in noisy multi-speaker chats","Speech model gains 93% accuracy on whether to reply","AI learns to stay silent when conversation isn't for it","Cocktail-Talker: AI that knows when to keep quiet","GRPO sharpens AI's turn-taking in noisy social audio"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000439,"raw_usage":{"total_tokens":2073,"prompt_tokens":762,"completion_tokens":1311,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":506,"completion_tokens_details":{"reasoning_tokens":1236}},"tokens_in":506,"tokens_out":1311,"duration_ms":10016,"temperature":1.0,"reasoning_tokens":1236,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T01:52:39.838906+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have human annotators listen to a sample of the synthetic scenes and mark whether the assistant should respond, remain silent but attentive, or ignore; compare their labels with the model's actions. If human agreement with the model is near chance, or if retraining on human labels produces similar accuracy numbers, the current results reflect label self-consistency rather than socially appropriate turn-taking.","supporting_citations":[],"review_version":1}