{"id":"7662ffed-3a7a-4de6-9bdf-4bd74c89fc76","arxiv_id":"2607.20460","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Current full-duplex speech systems largely fail to follow explicit turn-taking instructions: the best of six models scores 64.4% adherence on the new Instruct-FD benchmark, and proactive behaviors like interruption and backchanneling are especially weak.","lead":"This paper introduces Instruct-FD, a benchmark that checks whether full-duplex voice assistants can follow explicit turn-taking instructions such as 'backchannel when the user hesitates' or 'interrupt on contradictions'. Across six leading systems, the best model obeys the instruction only 64% of the time, revealing a large control gap in current spoken dialogue AI.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 64.4% IAS headline is only as trustworthy as the LLM judge, which is validated on 180 cases with 88.9% accuracy — and is least reliable on exactly the proactive Backchannel/Interrupt scores the paper emphasizes; no sensitivity analysis shows the numbers survive ASR timestamp error or the hard 2s","rationale":"The reader's conditional verdict is appropriate; my read does not move it. The most load-bearing assumption is that IAS scores measure model capability rather than judge/ASR artifacts. That assumption is weakest for Backchannel and Interrupt, which are also the main qualitative findings. Because the judge's validation has very sparse positives for those classes (1–2 TP), the reported low scores are statistically fragile. The paper's own A.5 attributes most errors to ASR transcript artifacts, and A.11's strict 2s threshold is hand-set; no robustness analysis is given. A release of code/data would allow the sensitivity test, but even without release the authors could run it internally. Until then, the central claim should remain conditional. Note the human study is real supporting evidence for test-case actionability, but it does not close the judge-reliability gap; if anything, the 63.7% human backchannel accuracy shows how hard even human judgment is for the timing-sensitive proactive cases.","tokens_in":25329,"tokens_out":7822,"duration_ms":69277,"concrete_test":"Recompute Gemini's IAS (and the other five models) after two perturbations: (a) rerun the judge on all 912 cases with Qwen3 word timestamps shifted by ±200 ms and ±500 ms within overlapping regions; (b) re-run with the yield/continue cutoff in A.11 set to 1.5 s and 2.5 s instead of 2.0 s. If overall IAS changes by more than 5 percentage points, or Backchannel/Interrupt scores change by more than 10 points, or the best-model ranking flips, the headline gap is not robust to the ASR-timestamp/threshold assumptions. As an alternative high-cost check, have two independent annotators judge a stratified sample of 300 model-response WAVs directly from audio (no ASR transcript) and compare the resulting IAS to the LLM-judge IAS.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim — that current FD systems show a 'substantial gap' in instruction-following turn management, with best IAS 64.4% — rests entirely on LLM-judge verdicts. The judge is validated on only 180 cases (88.9% accuracy), and the validation is thinnest precisely where the paper's headline findings live: Backchannel F1=67.3 (TP=1, FN=2, FP=1, TN=26) and Interrupt F1=75.7 (TP=2, FN=2, FP=1, TN=25) in Appendix A.5. With so few positive examples, the near-zero to 50% scores on these behaviors could shift substantially with a modest number of judge errors. The judge's timing classifications further depend on Qwen3 ASR/forced-alignment word boundaries and on hard-coded thresholds — the '2s is STRICT and NON-NEGOTIABLE' yield/continue cutoff in A.11, '>4 words' overlap = interruption, backchannel '<4 words'. The paper reports no sensitivity analysis perturbing timestamps or varying these cutoffs. The human study (Section 5) validates that test cases are natural and instructions actionable, but it does not validate the judge on actual model responses at scale; indeed A.13 concedes even human backchannel accuracy is 63.7% due to annotation/perception mismatch. Thus the reported 64.4% and the per-instruction gaps are not yet robust to the measurement pipeline.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"Instruct-FD addresses whether full-duplex spoken dialogue systems can follow explicit turn-taking instructions. The paper contributes a benchmark generation pipeline: LLM-generated scenario-conditioned conversations with inline markers for overlap events, TTS synthesis with forced alignment, a WebRTC-style orchestrator that streams user audio and records two channels, and an LLM judge that reconstructs a timestamped transcript and decides instruction adherence. Six models are evaluated under five instruction types (Backchannel, Listen, Interrupt, Continue, Acknowledge) across 29 scenarios. The headline result is that the best model (Gemini-Live) reaches 64.4% overall IAS, with proactive behaviors (Backchannel 0–50%, Interrupt 1.7–11.7%) far weaker than responsive ones. A human study (11 participants, 89.6% case accuracy) is used to validate that test cases are natural and instructions actionable, and a 180-case validation of the LLM judge reports 88.9% accuracy.","tokens_in":25749,"tokens_out":5413,"duration_ms":45966,"significance":"The benchmark targets a genuine and underexplored capability: controllable turn management. The design has notable strengths: the same-conversation/different-instruction setup controls for content; ΔIAS is a sensible sensitivity metric; the human study gives evidence of naturalness and actionability; and the appendix documentation is unusually detailed. If the measurement pipeline is robust, the paper would be a useful resource for the FD dialogue community. However, the empirical conclusions are only as strong as the judge, and the current validation is thinnest for exactly the proactive behaviors the paper emphasizes.","major_comments":[{"comment":"The judge validation contains only 1 and 2 true positives for Backchannel and Interrupt, respectively (TP/FN/FP/TN: 1/2/1/26 and 2/2/1/25). Accuracy is therefore dominated by the large negative class. Given the paper's central claim that proactive behaviors are particularly challenging, the reader cannot tell whether the near-zero Interrupt scores and low Backchannel scores reflect model failures or judge uncertainty. Please report confidence intervals for per-instruction F1, and/or re-validate with a set that oversamples positive events; at minimum, show how per-instruction IAS changes if judge error rates at the level suggested by the confusion matrix are applied.","section":"A.5, Table 23; Section 2.4"},{"comment":"The judge applies hard thresholds (2s yield/continue cutoff, >4 words overlap = interruption, <4 words = backchannel) to word-level timestamps obtained from Qwen3 ASR/forced alignment. The orchestration also injects barge-ins at 3s after model speech onset. No sensitivity analysis is provided for either timestamp noise or threshold values. A systematic 100–200ms ASR timing drift could move events across the 2s boundary and change category-level IAS substantially. The authors should report a threshold sweep (e.g., 1–3s) and/or an ASR timestamp perturbation study, and justify the 'strict and non-negotiable' 2s rule empirically.","section":"A.11; Section 2.4; Section 3.1"},{"comment":"Human backchannel accuracy is only 63.7%, and the appendix attributes part of this to a mismatch between offline LLM-selected ground-truth timestamps and online human perception. If human listeners cannot identify the prescribed backchannel moments, then the benchmark's backchannel labels may not reflect what is perceptually natural in real time, making it hard to interpret low model Backchannel scores as a capability gap. This concern should be addressed by either re-anchoring backchannel ground truth to perceptually validated moments or by relaxing the timing tolerance and re-reporting scores.","section":"A.13; Section 5"}],"minor_comments":[{"comment":"Typos: 'traditinal', 'improvment'; title has 'T urn-taking' spacing issue.","section":"A.1.1"},{"comment":"The appendix refers to the 'Duplex-IF benchmark'; the name elsewhere is Instruct-FD.","section":"A.7"},{"comment":"Incomplete sentence: 'its response is .' should be completed.","section":"A.9.3"},{"comment":"Spacing issue: 'Inbarge-inturns' — also specify whether the 3s barge-in delay is measured from model speech onset or from another reference point.","section":"Section 3.1"},{"comment":"The '2s limit is STRICT and NON-NEGOTIABLE' is asserted without empirical justification or citation; please add a rationale or reference.","section":"A.11"},{"comment":"State how Overall IAS is computed: unweighted over all test cases, or weighted by instruction group? This matters because instruction groups have very different test counts (e.g., Listen 312 vs Interrupt 120).","section":"Table 3"},{"comment":"The ±2s backchannel tolerance appears only in the appendix; the main-text judge description should state the tolerance explicitly.","section":"A.13 / Section 2.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is potentially valuable and the human validation of the benchmark itself is a strength. The main risk is that the headline empirical results may not survive careful stress-testing of the judge; however, this is addressable with added analyses rather than a redesign. I do not see evidence of circularity or dishonest reporting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Good paper to know about. The pitch is exactly right: current FD benchmarks measure turn-taking quality against fixed priors, but nobody tests whether the model can change its behavior when told to. Instruct-FD does that with a same-conversation-different-instruction design, and the synthetic pipeline with inline overlap markers is clever. The human study (11 participants, 89.6% accuracy) is real evidence that the test cases are natural and the instructions actionable. The orchestrator that runs the same audio against six heterogeneous models through public interfaces is also a solid engineering contribution. This is a new evaluation axis, not a repackaging.\n\nThe soft spot is the measurement pipeline behind the headline numbers. The LLM judge is validated on 180 cases at 88.9%, which is plausible overall, but the validation is thinnest exactly where the findings live: Backchannel has 1 true positive and 2 false negatives, Interrupt has 2 true positives. So the judge's accuracy on the proactive behaviors—where models score near zero—is essentially unmeasured. The judge also relies on hard-coded 2-second yield/continue cutoffs and ASR word timestamps, with no sensitivity analysis. A modest ASR timing error or a different threshold could move the per-category scores, and possibly the 64.4% headline, by a noticeable amount. The qualitative case studies in A.9 show real failures, so the qualitative direction is probably right; but the precise numbers should be read with caution. Also, the benchmark data and code have not been released, which is limiting for a benchmark paper.\n\nThe delta-IAS analysis is a nice addition, though with the same measurement caveats. The paper is honest about its limits (e.g., A.13 on backchannel annotation mismatch), which I take as a good sign.\n\nI'd send this to peer review. The core idea is reusable and the benchmark likely becomes a reference point for FD evaluation. The authors should be asked to release the data, expand judge validation on proactive categories, and run a threshold/timestamp sensitivity analysis before the numbers are treated as definitive.","headline":"A genuinely new benchmark for turn-taking as instruction-following, with a real gap in proactive control—but the headline numbers depend on an under-powered judge and unreleased data.","tokens_in":26293,"tokens_out":2517,"would_cite":true,"duration_ms":22991,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Instruction-following turn management is largely missing in full-duplex speech systems: the best of six obeys explicit turn-taking instructions only 64.4% of the time, and proactive behaviors lag far behind.","keywords":["full-duplex speech","turn-taking","instruction following","benchmark","LLM judge","backchannel","interruption","spoken dialogue"],"falsifier":"Re-score the full benchmark using a different forced aligner or human-verified word boundaries and compare IAS. A material swing in the best model's 64.4% — or movement of Gemini's Backchannel score across the 2s continue/yield threshold — would show the reported gap is partly a timing-measurement artifact rather than model behavior.","tokens_in":25239,"feed_emoji":"🎙️","tokens_out":5844,"duration_ms":48393,"temperature":0.7,"pith_summary":"The paper introduces Instruct-FD, a benchmark that treats turn management in full-duplex spoken dialogue — systems that can listen and speak simultaneously — as an instruction-following problem. It generates synthetic conversations with scripted overlap events, runs them through six state-of-the-art systems using a deployment-agnostic orchestrator, and judges adherence with a validated LLM judge. The headline result is a large capability gap: the best system follows instructions 64.4% of the time, with proactive behaviors (backchanneling during user speech, interrupting on a trigger) far worse than responsive ones (continuing or yielding after a user interjection). The paper also reports instruction sensitivity, showing that some systems genuinely change behavior when instructed while others barely shift. The upshot: controllable turn-taking is a bottleneck for deploying one conversational system across applications with different policies.","feed_headline":"Full-duplex AI obeys turn-taking instructions just 64.4%","feed_subtitle":"Benchmark of six speech systems finds proactive backchanneling and interruption lag far behind passive listening.","key_machinery":"The central mechanism is the instruction-conditioned test case: one synthetic conversation is paired with several turn-management instructions (backchannel, interrupt, listen, continue, acknowledge), each yielding a different valid behavior, isolating adherence from content. A multi-turn orchestrator streams scripted user audio, including scripted barge-ins, and records a time-aligned two-channel WAV. An LLM judge aligns word-level timestamps from ASR and forced alignment, builds an interleaved transcript, and applies strict timing rules (e.g., a 2-second cutoff for continue vs. yield) to produce pass/fail verdicts. Two metrics follow: Instruction Adherence Score (IAS) and Instruction Sensit","core_discovery":"Current full-duplex systems cannot reliably modulate turn-taking in response to explicit instructions, despite human-validated test cases that are natural and actionable. On 912 instruction-conditioned test cases, the best of six systems reaches 64.4% adherence. Continue is strong (up to 97.5%) while Backchannel (0–50%) and Interrupt (1.7–11.7%) are consistently poor. An instruction-sensitivity metric shows the biggest induced shift is +42.5% (Gemini on Backchannel), and one model drops 11.2% on Acknowledge when instructed, suggesting prompting can destabilize behavior. The paper concludes the gap reflects model capability, supported by human accuracy of 89.6% on the same cases.","pith_inferences":["The paper's 'semantic compliance without temporal compliance' failure mode suggests that next-token training is insufficient; timing-aware rewards (e.g., reinforcement learning that penalizes post-turn acknowledgments) are a natural, untested extension.","The orchestrator's real-time control could support adaptive test cases — branching user audio that reacts to model behavior — which would relax the two-turn constraint the paper explicitly flags as a limitation.","The taxonomy could be composed into instructions like 'interrupt on contradictions but otherwise listen silently' to probe whether models can chain multiple policy conditions in one instruction.","Because turn-taking conventions vary across languages, the English-only benchmark likely underestimates or overestimates capabilities for other languages; re-running with multilingual TTS and human validation would be a direct extension."],"forward_implications":["Controllable turn management is a distinct, largely missing capability: even the best system fails more than a third of explicit turn-taking instructions, so gains in fluency have not translated into controllability.","Proactive behaviors — backchanneling at the right moment and interrupting on a trigger — are the binding constraint; responsive behaviors like continuing through a backchannel are already near-ceiling for several models.","The same-conversation/different-instruction design gives a direct training and evaluation signal: a model that follows instructions will change its turn behavior on identical audio, which can be measured with ΔIAS.","Instruction sensitivity varies widely (from −11.2% to +42.5%), so aggregate quality scores hide whether a system actually responds to policy; both IAS and ΔIAS should be reported.","Deploying one system across applications with different conversational policies (e.g., proactive tutoring vs. passive counseling) remains impractical until this gap narrows."],"fun_headline_variants":["Instruction-following turn-taking: best AI only 64.4% compliant","Full-duplex AI can't follow turn-taking commands—best scores 64.4%","AI turn-taking obeys instructions poorly: top model 64.4%","Even best full-duplex AI only 64.4% on turn-taking instructions","Turn-taking instruction gap: AI backchanneling and interruption fail"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The headline numbers rest on the LLM judge's timing verdicts, which depend on ASR word timestamps and a strict 2-second cutoff; the judge was validated against humans on only 180 of 912 cases (88.9% agreement), so if those timestamps are off near the cutoffs, every adherence score — including 64.4% — could shift materially.","fun_headline_variants_meta":{"raw":{"variants":["Instruction-following turn-taking: best AI only 64.4% compliant","Full-duplex AI can't follow turn-taking commands—best scores 64.4%","AI turn-taking obeys instructions poorly: top model 64.4%","Even best full-duplex AI only 64.4% on turn-taking instructions","Turn-taking instruction gap: AI backchanneling and interruption fail"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000699,"raw_usage":{"total_tokens":2979,"prompt_tokens":717,"completion_tokens":2262,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":461,"completion_tokens_details":{"reasoning_tokens":2159}},"tokens_in":461,"tokens_out":2262,"duration_ms":14010,"temperature":1.0,"reasoning_tokens":2159,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T13:53:27.312148+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-score the full benchmark using a different forced aligner or human-verified word boundaries and compare IAS. A material swing in the best model's 64.4% — or movement of Gemini's Backchannel score across the 2s continue/yield threshold — would show the reported gap is partly a timing-measurement artifact rather than model behavior.","supporting_citations":[],"review_version":1}