Pith. sign in

REVIEW 3 major objections 7 minor 15 references

Instruction-following turn management is largely missing in full-duplex speech systems: the best of six obeys explicit turn-taking instructions only 64.4% of the time, and proactive behaviors lag far behind.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 13:53 UTC pith:C3AUE76H

load-bearing objection A genuinely new benchmark for turn-taking as instruction-following, with a real gap in proactive control—but the headline numbers depend on an under-powered judge and unreleased data. the 3 major comments →

arxiv 2607.20460 v1 pith:C3AUE76H submitted 2026-05-15 cs.CL cs.AI

Instruct-FD: Can Your Full-Duplex Speech System Follow Turn-Taking Instructions?

Yuzhi Tang , Wentao Ma , Xiling Zhao , Ahmad Salimi , Sepehr Harfi Moridani , Dongming Shen , Jixuan Wang , Abdulrahman Abdulrazzag
show 12 more authors
Murdock Aubry Yu-Hua Chen Daniel Lee Jaewon Lee Jonah Mackey Silin Meng Nicholas Stranges Chenxu Xiong Hao Yu Yi Zhu Mu Li Alex Smola
This is my paper
classification cs.CL cs.AI
keywords full-duplex speechturn-takinginstruction followingbenchmarkLLM judgebackchannelinterruptionspoken dialogue
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces Instruct-FD, a benchmark that treats turn management in full-duplex spoken dialogue — systems that can listen and speak simultaneously — as an instruction-following problem. It generates synthetic conversations with scripted overlap events, runs them through six state-of-the-art systems using a deployment-agnostic orchestrator, and judges adherence with a validated LLM judge. The headline result is a large capability gap: the best system follows instructions 64.4% of the time, with proactive behaviors (backchanneling during user speech, interrupting on a trigger) far worse than responsive ones (continuing or yielding after a user interjection). The paper also reports instruction sensitivity, showing that some systems genuinely change behavior when instructed while others barely shift. The upshot: controllable turn-taking is a bottleneck for deploying one conversational system across applications with different policies.

Core claim

Current full-duplex systems cannot reliably modulate turn-taking in response to explicit instructions, despite human-validated test cases that are natural and actionable. On 912 instruction-conditioned test cases, the best of six systems reaches 64.4% adherence. Continue is strong (up to 97.5%) while Backchannel (0–50%) and Interrupt (1.7–11.7%) are consistently poor. An instruction-sensitivity metric shows the biggest induced shift is +42.5% (Gemini on Backchannel), and one model drops 11.2% on Acknowledge when instructed, suggesting prompting can destabilize behavior. The paper concludes the gap reflects model capability, supported by human accuracy of 89.6% on the same cases.

What carries the argument

The central mechanism is the instruction-conditioned test case: one synthetic conversation is paired with several turn-management instructions (backchannel, interrupt, listen, continue, acknowledge), each yielding a different valid behavior, isolating adherence from content. A multi-turn orchestrator streams scripted user audio, including scripted barge-ins, and records a time-aligned two-channel WAV. An LLM judge aligns word-level timestamps from ASR and forced alignment, builds an interleaved transcript, and applies strict timing rules (e.g., a 2-second cutoff for continue vs. yield) to produce pass/fail verdicts. Two metrics follow: Instruction Adherence Score (IAS) and Instruction Sensit

Load-bearing premise

The headline numbers rest on the LLM judge's timing verdicts, which depend on ASR word timestamps and a strict 2-second cutoff; the judge was validated against humans on only 180 of 912 cases (88.9% agreement), so if those timestamps are off near the cutoffs, every adherence score — including 64.4% — could shift materially.

What would settle it

Re-score the full benchmark using a different forced aligner or human-verified word boundaries and compare IAS. A material swing in the best model's 64.4% — or movement of Gemini's Backchannel score across the 2s continue/yield threshold — would show the reported gap is partly a timing-measurement artifact rather than model behavior.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Controllable turn management is a distinct, largely missing capability: even the best system fails more than a third of explicit turn-taking instructions, so gains in fluency have not translated into controllability.
  • Proactive behaviors — backchanneling at the right moment and interrupting on a trigger — are the binding constraint; responsive behaviors like continuing through a backchannel are already near-ceiling for several models.
  • The same-conversation/different-instruction design gives a direct training and evaluation signal: a model that follows instructions will change its turn behavior on identical audio, which can be measured with ΔIAS.
  • Instruction sensitivity varies widely (from −11.2% to +42.5%), so aggregate quality scores hide whether a system actually responds to policy; both IAS and ΔIAS should be reported.
  • Deploying one system across applications with different conversational policies (e.g., proactive tutoring vs. passive counseling) remains impractical until this gap narrows.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper's 'semantic compliance without temporal compliance' failure mode suggests that next-token training is insufficient; timing-aware rewards (e.g., reinforcement learning that penalizes post-turn acknowledgments) are a natural, untested extension.
  • The orchestrator's real-time control could support adaptive test cases — branching user audio that reacts to model behavior — which would relax the two-turn constraint the paper explicitly flags as a limitation.
  • The taxonomy could be composed into instructions like 'interrupt on contradictions but otherwise listen silently' to probe whether models can chain multiple policy conditions in one instruction.
  • Because turn-taking conventions vary across languages, the English-only benchmark likely underestimates or overestimates capabilities for other languages; re-running with multilingual TTS and human validation would be a direct extension.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. Instruct-FD addresses whether full-duplex spoken dialogue systems can follow explicit turn-taking instructions. The paper contributes a benchmark generation pipeline: LLM-generated scenario-conditioned conversations with inline markers for overlap events, TTS synthesis with forced alignment, a WebRTC-style orchestrator that streams user audio and records two channels, and an LLM judge that reconstructs a timestamped transcript and decides instruction adherence. Six models are evaluated under five instruction types (Backchannel, Listen, Interrupt, Continue, Acknowledge) across 29 scenarios. The headline result is that the best model (Gemini-Live) reaches 64.4% overall IAS, with proactive behaviors (Backchannel 0–50%, Interrupt 1.7–11.7%) far weaker than responsive ones. A human study (11 participants, 89.6% case accuracy) is used to validate that test cases are natural and instructions actionable, and a 180-case validation of the LLM judge reports 88.9% accuracy.

Significance. The benchmark targets a genuine and underexplored capability: controllable turn management. The design has notable strengths: the same-conversation/different-instruction setup controls for content; ΔIAS is a sensible sensitivity metric; the human study gives evidence of naturalness and actionability; and the appendix documentation is unusually detailed. If the measurement pipeline is robust, the paper would be a useful resource for the FD dialogue community. However, the empirical conclusions are only as strong as the judge, and the current validation is thinnest for exactly the proactive behaviors the paper emphasizes.

major comments (3)
  1. [A.5, Table 23; Section 2.4] The judge validation contains only 1 and 2 true positives for Backchannel and Interrupt, respectively (TP/FN/FP/TN: 1/2/1/26 and 2/2/1/25). Accuracy is therefore dominated by the large negative class. Given the paper's central claim that proactive behaviors are particularly challenging, the reader cannot tell whether the near-zero Interrupt scores and low Backchannel scores reflect model failures or judge uncertainty. Please report confidence intervals for per-instruction F1, and/or re-validate with a set that oversamples positive events; at minimum, show how per-instruction IAS changes if judge error rates at the level suggested by the confusion matrix are applied.
  2. [A.11; Section 2.4; Section 3.1] The judge applies hard thresholds (2s yield/continue cutoff, >4 words overlap = interruption, <4 words = backchannel) to word-level timestamps obtained from Qwen3 ASR/forced alignment. The orchestration also injects barge-ins at 3s after model speech onset. No sensitivity analysis is provided for either timestamp noise or threshold values. A systematic 100–200ms ASR timing drift could move events across the 2s boundary and change category-level IAS substantially. The authors should report a threshold sweep (e.g., 1–3s) and/or an ASR timestamp perturbation study, and justify the 'strict and non-negotiable' 2s rule empirically.
  3. [A.13; Section 5] Human backchannel accuracy is only 63.7%, and the appendix attributes part of this to a mismatch between offline LLM-selected ground-truth timestamps and online human perception. If human listeners cannot identify the prescribed backchannel moments, then the benchmark's backchannel labels may not reflect what is perceptually natural in real time, making it hard to interpret low model Backchannel scores as a capability gap. This concern should be addressed by either re-anchoring backchannel ground truth to perceptually validated moments or by relaxing the timing tolerance and re-reporting scores.
minor comments (7)
  1. [A.1.1] Typos: 'traditinal', 'improvment'; title has 'T urn-taking' spacing issue.
  2. [A.7] The appendix refers to the 'Duplex-IF benchmark'; the name elsewhere is Instruct-FD.
  3. [A.9.3] Incomplete sentence: 'its response is .' should be completed.
  4. [Section 3.1] Spacing issue: 'Inbarge-inturns' — also specify whether the 3s barge-in delay is measured from model speech onset or from another reference point.
  5. [A.11] The '2s limit is STRICT and NON-NEGOTIABLE' is asserted without empirical justification or citation; please add a rationale or reference.
  6. [Table 3] State how Overall IAS is computed: unweighted over all test cases, or weighted by instruction group? This matters because instruction groups have very different test counts (e.g., Listen 312 vs Interrupt 120).
  7. [A.13 / Section 2.4] The ±2s backchannel tolerance appears only in the appendix; the main-text judge description should state the tolerance explicitly.

Circularity Check

0 steps flagged

No circularity: the benchmark, judge, and metrics are defined independently of the models' responses; the central empirical claim rests on an external (human-validated) measurement pipeline.

full rationale

The paper's derivation chain is not circular. IAS (Eq. 1) is defined as the LLM judge's pass rate over test cases; the judge is an external LLM prompted with the instruction, a temporally grounded transcript, and a ground-truth reference, and it is validated against 180 human annotations (88.9% accuracy, Appendix A.5). The generation pipeline (Section 2.2) creates test cases with scripted barge-in markers and recovers timestamps via ASR/forced alignment, independently of the models being tested. Nothing in the metric is fitted to model outputs; the only tuned component is the judge's hand-written threshold (e.g., '2s is STRICT and NON-NEGOTIABLE' in Appendix A.11), which is an evaluation policy, not a parameter learned from the data. The human study (Section 5) validates naturalness/actionability of cases, not the model scores, and the paper acknowledges its own limitations (Appendix A.13: backchannel annotation mismatch; Section 4: judge errors on ASR artifacts). These are measurement-validity concerns, not circularity. The only self-citation with author overlap (Tay et al., 2026, cited for ASR in A.1.1) is peripheral and not load-bearing. Therefore no step reduces to its own input; score 0.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The benchmark introduces no pseudo-physical entities. It does introduce hand-set evaluation constants (barge-in delay, VAD thresholds, judge timing cutoffs) that are not fitted to data but directly influence every reported score. The four axioms listed are the load-bearing premises for trusting the benchmark's measurements as a proxy for real-world turn-taking controllability.

free parameters (5)
  • barge-in delay (barge_in_after_s) = 3.0 s after model speech onset
    Sets the timing of user overlap in all barge-in turns; hand-chosen and directly shapes Interrupt/Acknowledge/Continue test difficulty.
  • VAD silence threshold = 1.5 s default (2.5 s Moshi, 5.0 s PersonaPlex/FunAudioChat)
    Defines when a turn is considered complete and triggers the next user segment; affects all proactive and responsive measurements.
  • judge continue/yield response-time threshold = 2 s
    LLM judge treats any pause >2 s after a user backchannel as yielding; a strict hand-set constant that strongly affects Continue scores.
  • judge interruption word-count threshold = >4 words during overlap = interruption
    Threshold for classifying model speech as interruption rather than backchannel; affects Backchannel and Interrupt metrics.
  • backchannel ground-truth tolerance = ±2 s around LLM-selected timestamps
    Tolerance window for human and model backchannels; acknowledged in A.13 as causing annotation mismatch with human perception.
axioms (4)
  • domain assumption Turn-taking can be decomposed into the five instruction categories (Backchannel, Interrupt, Listen, Continue, Acknowledge)
    The benchmark's coverage depends on this taxonomy being the right way to slice turn management; other important behaviors may be omitted.
  • domain assumption LLM judge adherence definitions (interrupt via reference comparison, 2 s continue limit, etc.) capture user-relevant turn-taking quality
    The judge protocol in Appendix A.11 bakes in specific timing criteria; the human study validates that cases are interpretable, but not that the judge's strict thresholds exactly match user perception for model outputs.
  • domain assumption The synthetic conversational scenarios and generated audio are sufficiently natural and representative of real FD interactions
    All test cases are generated from LLM turn agents and Gemini TTS; human validation (89.6% accuracy) supports interpretability, but distributional representativeness is not proven.
  • domain assumption Word-level timestamps from Qwen3-ASR and forced aligner are accurate enough for the judge's strict timing rules
    The judge relies on these timestamps to decide whether a backchannel occurred during user speech or a continuation pause exceeded 2 s; no timestamp-accuracy evaluation is provided.

pith-pipeline@v1.3.0-alltime-deepseek · 25140 in / 10154 out tokens · 77779 ms · 2026-08-02T13:53:27.312148+00:00 · methodology

0 comments
read the original abstract

Current full-duplex (FD) spoken dialogue systems can produce fluid interactions, yet it remains unclear whether they can adapt their turn-taking behavior when explicitly instructed. This is critical for real-world deployment, where conversational policies vary across applications (e.g., proactive tutoring vs. passive counseling). We introduce Instruct-FD, an instruction-conditioned benchmark for evaluating controllable turn management in FD systems. To enable this, we develop a human-validated, scalable synthetic pipeline that generates instruction-conditioned conversations, along with a deployment-agnostic multi-turn evaluation protocol and an LLM-based judge. Benchmarking six state-of-the-art full-duplex systems reveals a substantial gap in instruction-following turn management: the best model achieves only 64.4% adherence. Performance is highly uneven across behaviors and scenarios, with proactive behaviors such as model backchanneling and interruption remaining particularly challenging. These findings establish instruction-following turn management as a crucial direction for building adaptable and deployable full-duplex dialogue systems.

Figures

Figures reproduced from arXiv: 2607.20460 by Abdulrahman Abdulrazzag, Ahmad Salimi, Alex Smola, Chenxu Xiong, Daniel Lee, Dongming Shen, Hao Yu, Jaewon Lee, Jixuan Wang, Jonah Mackey, Mu Li, Murdock Aubry, Nicholas Stranges, Sepehr Harfi Moridani, Silin Meng, Wentao Ma, Xiling Zhao, Yi Zhu, Yu-Hua Chen, Yuzhi Tang.

Figure 1
Figure 1. Figure 1: Overview of the INSTRUCT-FD evaluation protocol. Left is the proactive be￾haviours, and right is the responsive behaviours. We present INSTRUCT-FD, a benchmark that pairs user audio streams affording specific turn-management opportunities with explicit natural-language instructions. We evaluate two complementary situations: (1) when the user is speaking (should the model remain silent, give a short acknowl… view at source ↗
Figure 2
Figure 2. Figure 2: The four steps test case generation pipeline. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Multi-turn user orchestrator protocol. A turn manager streams scripted user audio [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Opening page of the onboarding tutorial. Participants are introduced to the study [PITH_FULL_IMAGE:figures/full_fig_p030_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

15 extracted references · 2 linked inside Pith

  1. [1]

    Read the persona and pick exactly ONE concrete detail that is relevant to the scenario (e.g., name, a specific piece of personal information, occupation, location, hobby)

  2. [2]

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al

    URLhttps://arxiv.org/abs/2411.00774. Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone.arXiv preprint arXiv:2408.01800, 2024. He Zhang, Wenqian Cui, Haoning Xu, Xiaohui Li, Lei Zhu, Haoli Bai, Shaohua Ma, and Irwin King. Mtr-duplexbench: To...

    Pith/arXiv arXiv 2024

  3. [3]

    The sentence should convey what the scenario is about while grounding it in the persona | but stay broad enough that the turn-by-turn simulation can fill in the specifics

  4. [4]

    Do NOT include behavioral instructions

    Do NOT list multiple persona traits. Do NOT include behavioral instructions. Just one descriptive sentence

  5. [5]

    Fuse that detail with the scenario description into a single, broad sentence

  6. [8]

    A user dictates structured sequential information (e.g., a phone number or address) in natural chunks with pauses

    Faithfulness: do NOT invent new details. Use only what is in the persona and scenario. Examples: Scenario: "A user dictates structured sequential information (e.g., a phone number or address) in natural chunks with pauses" Persona detail: "Pedro, lives at 8847 Hillside Court, Rancho Cucamonga" →"Pedro dictates his home address to the assistant in natural ...

  7. [9]

    A behavioral instruction the MODEL was supposed to follow

  8. [10]

    A conversation transcript with word-level timestamps for both USER and MODEL

  9. [11]

    Expected Turn-Taking Behavior

    An "Expected Turn-Taking Behavior" reference showing what the \ MODEL should have done. == Transcript format == - The transcript has NO punctuation (it comes from speech recognition). Infer \ punctuation from context. Phrases like "right", "you know", "okay", "you there" \ at clause boundaries are likely questions. - Lines marked **[Overlap]** indicate si...

  10. [12]

    Locate the USER turn in question in the transcript

  11. [13]

    Find the SAME USER turn in the groundtruth reference | the reference shows \ what the full, uninterrupted USER turn would look like

  12. [14]

    Compare the transcript to the reference: how much of the full USER turn had \ been delivered at the point the MODEL started speaking?

  13. [15]

    no" and stop.) 1a. The MODEL must have at least some speech in the transcript. \ Zero MODEL words→

    Decision: - If the USER had NOT yet delivered the full content (significant words \ from the reference remain unspoken in the transcript)→the MODEL DID \ interrupt. This is true even if there is no timestamp overlap | the \ USER stopping short of their full reference turn is itself evidence \ that the MODEL's speech caused the USER to stop. - If the USER ...

  14. [2024]

    Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou

    URLhttps://arxiv.org/abs/2410.17196. Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. Qwen-Audio: Advancing Audio-Language Models, 2023. URL https://arxiv.org/abs/2311.07919. Alexandre D´efossez, Laurent Mazar ´e, Manu Orsini, Am ´elie Royer, Patrick P´erez, Herv´e J´egou, Edouard Grave, and Neil Zegh...

    Pith/arXiv arXiv 2023

  15. [2025]

    you know?

    uses a turn-taking prediction model trained on the Switchboard dataset to evaluate transition timing. A second line of work extends evaluation to(2) model-initiated behaviors during user speech, such as backchanneling and proactive interruption. FLEXI (Ge et al., 2025) expanded coverage for model interruption, though its evaluation is scoped primarily to ...

    2025