{"id":"c625d938-29b8-4194-86d4-e297b3223abb","arxiv_id":"2607.29125","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"M3-DuplexBench, a bilingual English/Japanese benchmark over chat and multi-turn QA with teacher-forced full-context evaluation, reveals clear model-, language-, and domain-dependent gaps in full-duplex spoken dialogue systems.","lead":"This paper introduces a benchmark for testing spoken dialogue AI systems that can talk and listen at the same time, in English and Japanese, across casual chat and question-answering. It matters because current tests don't fairly compare how such systems use conversation history, languages, or different dialogue types.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Full-context conditioning may leak reference turn timing into the model, so the timing findings could be artifacts; test by time-shifting previous system turns in the conditioning stream.","rationale":"The paper's central value is a controlled multi-turn protocol and the four empirical findings it supports. The Full condition is the novel part; if it leaks reference timing to the model, the timing findings (2 and 3) are confounded. This is not a mere philosophical objection: Table II shows the average SHIFT durations in the synthetic task data are roughly half those in the chat reference corpora, so the cross-domain timing comparison is entangled with the timing schedule built into the stimuli. The proposed time-shift experiment isolates the leakage channel cleanly. I do not think this requires rejection: the benchmark may still be useful for content evaluation, and the authors already hedge with the teacher-forced gap caveat. But the timing claims should be re-examined after this control, and the synthetic task-domain timing distribution needs explicit validation. This is consistent with the reader's CONDITIONAL verdict, so no verdict change; the condition is now more specific.","tokens_in":11808,"tokens_out":7874,"duration_ms":88525,"concrete_test":"Take a subset of SHIFT samples from the task-oriented English set. Create a 'timing-shifted Full' condition: delay the start time of every previous system turn in the conditioning stream by +300 ms (keeping audio content, user timing, and target event identical), and re-run the same models under Full. If median model latency at the target SHIFT increases by a substantial fraction of 300 ms, Full is directly controlling timing through the conditioning signal; if latency is essentially unchanged, the timing findings reflect content-driven adaptation. A complementary check would compare QA accuracy under Full with gold reference history vs. a free-rollout history; if accuracy drops, the 'history helps' claim is oracle-context only.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Full-context conditioning (Section IV-A) feeds both speaker channels in (C,P), so the model hears the exact onset/offset of every previous reference system turn. Moshi-style duplex models can condition on these inter-turn intervals. The Full-vs-None latency reductions (e.g., Moshi task T1: 2.453s -> T3: 0.430s) and the claim that task-oriented turn-taking is easier may therefore reflect imitation of a supplied timing schedule rather than intrinsic multi-turn timing ability. Table II heightens this concern: average SHIFT durations are 0.88s/0.72s for chat En/Ja but 0.45s/0.35s for task En/Ja, even though the task dialogues were synthesized using the same chat corpora as timing references. If the conditioning signal itself contains the reference gaps, the Full condition is not a neutral controlled context for timing; it leaks the answer to the timing question. The paper acknowledges a teacher-forced trajectory gap (Section V-B.1) but only for content/state, not this direct timing-leakage channel.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes M3-DuplexBench, a multi-turn, multilingual (English/Japanese), multidomain (chat and task-oriented QA) benchmark for full-duplex spoken dialogue systems. It defines four event types (turn shift, pause, backchannel, barge-in) and evaluates models under three context conditions: None, user-side-only, and teacher-forced full-context. Experiments with four Moshi-based models report timing and content metrics and claim that (1) dialogue history helps multi-turn QA, (2) models adjust response timing to dialogue context, (3) smooth turn-taking is easier in task-oriented than chat dialogues, and (4) Japanese models lag mainly in content rather than timing. The paper proposes a representative value averaging the None and Full conditions and compares models across languages and domains.","tokens_in":12002,"tokens_out":9094,"duration_ms":102458,"significance":"If the protocol is valid, M3-DuplexBench would be a useful, controlled multi-turn evaluation framework for FDSDSs, extending coverage to Japanese and to task-oriented QA with multi-turn context. The idea of comparing context conditions is thoughtful and the event extraction follows prior work. However, the current manuscript does not release code/data, and several load-bearing aspects of the evaluation—particularly the timing information implicitly provided by teacher-forced full-context conditioning and the unexplained timing statistics of the synthetic task dialogues—need to be addressed before the stated findings can be relied upon. The content evaluations also lack reliability evidence. With these concerns resolved, the benchmark could be a solid contribution to the community.","major_comments":[{"comment":"In the Full condition, the model conditions on the reference system speech stream, so it observes the exact onset/offset times of all previous system turns. Moshi-style models can use these inter-turn intervals to predict the current SHIFT timing. The large latency reductions under Full (e.g., Moshi T1: 2.453s -> T3: 0.430s) and the fact that Full latencies approach the dataset means in Table II are thus compatible with timing imitation rather than context-based adjustment. The paper's teacher-forced limitation statement does not address this channel. Please add a control that jitters/time-shifts the system turns in the conditioning stream; if latency follows the shifted schedule, the timing findings are artifacts.","section":"Section IV-A / V-B.1"},{"comment":"The synthetic task-oriented dialogues are built by sampling SHIFT/PAUSE/BC from the chat reference corpora, yet Table II reports average SHIFT durations of 0.45s (En) and 0.35s (Ja) for task dialogues vs 0.88s/0.72s for chat. This large, unexplained discrepancy means the cross-domain timing comparison is not controlled: task SHIFT events are intrinsically shorter, so Full-condition models will naturally show lower latency and higher TOR. Please explain the discrepancy, and either match SHIFT distributions across domains or validate the synthetic timing against natural task-oriented speech.","section":"Section III-B.2 / Table II / V-B.3"},{"comment":"All content metrics (relevance, consistency, QA accuracy) rely on a single GPT-5 nano judge, with no inter-judge agreement, no human validation, and no statement of whether the judge inputs are ASR transcripts or audio. Whisper errors on Japanese TTS output could differentially lower content scores, threatening the cross-lingual and context-effect claims (e.g., T8 vs T12). Report judge reliability on a sample, use multiple judges, and specify the judge input.","section":"Section IV-B.2"},{"comment":"The manuscript does not state whether the benchmark data, event extraction code, synthetic dialogue generation pipeline, or evaluation scripts will be released. As a benchmark contribution, this absence prevents reproduction and adoption. An availability statement with a public release is required for the contribution to be usable.","section":"Overall / Data Availability"}],"minor_comments":[{"comment":"The notation e=(ℓ,s,t) is not fully defined for SHIFT events with overlaps (t<s). Clarify how s and t relate to the speaker-change boundary and how the evaluation window W=[s,s+Δ] is anchored.","section":"Section III-A"},{"comment":"The 'None&Full' representative value is an unweighted average of two qualitatively different conditions. Provide a rationale or report the two conditions separately in summary tables.","section":"Section V-B.1 / Tables IV-V"},{"comment":"The text in the case-study figures is very small; consider providing the example transcripts in a readable table or enlarged font.","section":"Figures 2-3"},{"comment":"Reference [9] appears as both MTR-DuplexBench and MTRDuplexBench; standardize the spelling. Table I uses 'MTR' without defining it in the caption.","section":"References / Table I"},{"comment":"It would be helpful to state the model sizes or parameter counts for the four evaluated models to aid comparison.","section":"Section V-A"},{"comment":"Clarify what 'dur' represents (mean SHIFT duration? standard deviation?) and how it is computed from the generated dialogues.","section":"Table II"}],"recommendation":"major_revision","confidential_remarks":"The paper has a good central idea but currently overclaims the timing findings without a control for the timing information present in the Full-condition conditioning stream. The unexplained SHIFT duration gap between chat and synthetic task dialogues suggests the cross-domain comparison may be an artifact. With additional control experiments, reliability analyses, and a public release, the benchmark could be a solid contribution; without these, acceptance is risky."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: this is a real contribution to FDSDS evaluation, but the timing half of the story has a load-bearing confound that the authors don't address. The paper is still worth a serious referee, but it needs major revision.\n\nWhat's new: M3-DuplexBench is the first benchmark I know that combines multi-turn, English/Japanese, chat/task QA, and three context conditions in one protocol. The teacher-forced full-context condition applied to timing metrics is a step beyond MTR-DuplexBench. The content findings—dialogue history helps models answer multi-turn QA, and User context is unstable—are plausible and backed by the examples. The Japanese coverage fills a real gap.\n\nWhere it gets soft: the Full-context condition feeds the model both channels, including the exact onsets and offsets of the reference system turns. That means the model literally hears the timing schedule it's being tested on. So when Full reduces latency toward the dataset mean, \"models adjust their response timing to context\" may just mean they imitate the supplied schedule. The paper acknowledges a teacher-forced trajectory gap but only for content/state, not for this direct timing channel. The cross-domain claim is also entangled: Table II shows task SHIFT durations (0.45s/0.35s) are much shorter than chat (0.88s/0.72s), even though the text says task timing events are sampled from the same reference chat corpora. If the task dialogues actually use different timing stats, then the cross-domain timing comparison isn't clean. If they don't, the numbers need explaining.\n\nOther weaknesses are more standard: no code/data release for a benchmark is a problem; the synthetic task dialogues are unvalidated; the content scores rest on a single GPT-5 nano judge with no agreement check; only four Moshi-based models, so cross-lingual claims are tentative. The \"representative value\" averaging None and Full is ad hoc and should be justified or replaced.\n\nNone of this sinks the core idea. The controlled multi-turn protocol is genuinely useful, and the content results stand apart from the timing issue. But the timing conclusions need a control experiment—e.g., time-shifting the system turns in the conditioning stream—before I'd trust them.\n\nFor peer review: yes, send it out. A serious referee will do the authors good. The benchmark design is worth the field's attention, but the current claims about timing are overreaching. My recommendation is major revision.","headline":"Useful benchmark with a real comparability fix, but the Full-context timing results may partly measure the model copying the reference timing schedule rather than intrinsic multi-turn ability.","tokens_in":12543,"tokens_out":3771,"would_cite":true,"duration_ms":39873,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"M3-DuplexBench evaluates full-duplex spoken dialogue models under three context conditions and shows that supplying a coherent dialogue history improves both answer accuracy and turn-taking timing.","keywords":["full-duplex spoken dialogue","benchmark","turn-taking","multi-turn evaluation","teacher-forced conditioning","multilingual","task-oriented dialogue","chat"],"falsifier":"Run a free-running simulation where the model itself produces the system speech for the same dialogue, then evaluate at the same event; if timing and content scores diverge systematically from the teacher-forced condition (especially in chat), the full-context protocol is measuring an artifact of forced conditioning rather than natural multi-turn ability.","tokens_in":11673,"feed_emoji":"🗣️","tokens_out":4055,"duration_ms":41369,"temperature":0.7,"pith_summary":"This paper introduces M3-DuplexBench, a benchmark for full-duplex spoken dialogue systems that can listen while speaking. Its central claim is that multi-turn evaluation becomes fair and informative when every model is forced to condition on the same complete dialogue history, including the system's own previous speech. Under that protocol, the paper shows that dialogue history improves answer accuracy in multi-turn question answering, that models adapt their response timing to the rhythm of the context, that turn-taking is easier in task-oriented QA than in casual chat, and that Japanese full-duplex models' main gap is language understanding rather than timing.","feed_headline":"Dialogue history boosts full-duplex speech models' QA accuracy","feed_subtitle":"New English-Japanese benchmark isolates timing and content across chat and QA tasks.","key_machinery":"The key mechanism is teacher-forced full-context conditioning: before the target event, the model's state is driven by both user and system speech from a pre-recorded dialogue, so all models see the identical coherent history. This is applied to four event types (turn shift, pause, backchannel, barge-in) with timing and content metrics, and compared against single-turn (None) and user-history-only (User) conditions.","core_discovery":"M3-DuplexBench evaluates models under three context conditions — no history, user history only, and teacher-forced full history — and the paper's main discovery is that the full-context condition is the most reliable static setting, because it removes the context mismatch that plagues user-only conditioning. Using this protocol on Moshi-based models, the paper finds that dialogue history helps English models answer context-dependent questions, that models move their turn-taking latency closer to human reference timing when given full context, and that the Japanese-language models evaluated lag primarily in content generation, not in turn-taking behavior.","pith_inferences":["Because the full-context condition forces the model to follow a pre-recorded system trajectory, it may overestimate a model's ability to maintain coherent multi-turn state when the model would have chosen different wording; a natural extension is to check agreement between teacher-forced and free-running continuations on the same history.","The finding that User context often degrades timing suggests that earlier multi-turn benchmarks that used user-only conditioning may have understated models' true multi-turn timing ability; this could be tested by re-running those benchmarks with full-context conditioning.","The synthetic task-domain dialogues are built by sampling timing statistics from chat corpora; an untested assumption is whether chat timing transfers to QA, so the paper's cross-domain conclusions could be validated with human-negotiated QA timing samples.","The benchmark currently only supports Moshi-style architectures because of the parallel-stream conditioning requirement; if the protocol were adapted to cascaded models, it might reveal whether the cross-lingual content gap is specific to end-to-end systems."],"forward_implications":["If the full-context protocol is accepted, future FDSDS benchmarks can compare models on identical multi-turn histories rather than divergent simulated conversations.","Dialogue history should be treated as a genuine input feature in FDSDS evaluation, since it measurably changes both timing and content.","The observed cross-domain gap suggests that turn-taking evaluations should be domain-specific; chat and task-oriented QA place different demands on when to speak.","The cross-lingual finding points Japanese full-duplex development toward content generation and understanding rather than timing subroutines.","The paper's method for synthesizing task-oriented spoken dialogues from text QA corpora can be reused to build benchmark data for other languages."],"fun_headline_variants":["Full dialogue history sharpens spoken AI's QA accuracy","Dialogue memory fixes context mismatch in full-duplex models","Japanese spoken AI lags in content, not turn-taking timing","New multilingual benchmark exposes language gaps in spoken models","Full context makes spoken AI's turn-taking more human-like"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that teacher-forced full-context conditioning produces valid, natural model behavior at the evaluation window, even though the model's state is driven by pre-recorded system speech it did not generate itself.","fun_headline_variants_meta":{"raw":{"variants":["Full dialogue history sharpens spoken AI's QA accuracy","Dialogue memory fixes context mismatch in full-duplex models","Japanese spoken AI lags in content, not turn-taking timing","New multilingual benchmark exposes language gaps in spoken models","Full context makes spoken AI's turn-taking more human-like"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000135,"raw_usage":{"total_tokens":936,"prompt_tokens":658,"completion_tokens":278,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":402,"completion_tokens_details":{"reasoning_tokens":199}},"tokens_in":402,"tokens_out":278,"duration_ms":3481,"temperature":1.0,"reasoning_tokens":199,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T13:10:21.311314+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a free-running simulation where the model itself produces the system speech for the same dialogue, then evaluate at the same event; if timing and content scores diverge systematically from the teacher-forced condition (especially in chat), the full-context protocol is measuring an artifact of forced conditioning rather than natural multi-turn ability.","supporting_citations":[],"review_version":1}