{"id":"99d26f07-8e05-4457-afb6-35f706603104","arxiv_id":"2608.10878","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A streaming ASR model gains a parallel head that predicts turn states on the same 80 ms frame timeline as the transcript, improving bilingual turn-taking accuracy over a streaming baseline.","lead":"This paper adds a frame-by-frame 'turn state' predictor to a streaming speech recognizer, so a voice assistant can tell in real time whether you are done speaking, interrupting, or just saying 'um'. On Chinese and English turn-taking benchmarks it beats the main streaming baseline while keeping roughly the same latency.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported turn-state accuracies hinge on an undocumented mapping from EasyTurn's native labels to the five-state taxonomy; without that mapping, Table 1 does not establish the claimed superiority over SoulX-Duplug.","rationale":"The central claim is that X2-Turn delivers accurate frame-synchronous turn-state prediction at low latency. That claim is operationalized entirely through accuracy numbers on EasyTurn in Table 1. For those numbers to support the claim, the ground-truth labels used in evaluation must denote the same states the model is trained to predict. The paper shows the training side in detail: Qwen3.5-Plus annotates each word with one of the five states from Section 2.3, and ASR-anchored supervision projects those labels onto the 80 ms timeline. The evaluation side is left underspecified: EasyTurn (reference [8]) natively provides four turn states, but the paper does not state the mapping from those states to <|idle|>, <|noidle|>, <|incomplete|>, <|complete|>, and <|backchannel|>, nor how the 'last non-idle predicted state' comparison forms utterance-level ground truth. If the mapping is inconsistent with the LLM training labels, the model is scored against a different construct, and the Table 1 margins—especially the large Chinese ACC_comp gap (91.00 vs 77.67)—may be artifacts of label alignment rather than real turn-taking ability. This is not a matter of disagreement with field consensus; it is an internal evaluation-validity gap. I considered the ASR table issue (Stage2-Turn appears worse than Freeze-Omni on some sets despite the text claiming parity) and the missing error bars, but those concern secondary claims or precision, whereas the label mapping directly undermines the paper's central quantitative support. The reader's weakest_assumption identifies the same issue, and I agree. A conditional verdict remains appropriate, conditional on specifying and validating the label mapping, so I leave the reader's verdict unchanged.","tokens_in":8544,"tokens_out":5835,"duration_ms":52009,"concrete_test":"Obtain the EasyTurn label definitions and the authors' label-mapping table. On a held-out sample (e.g., 500 utterances from EasyTurn-zh), compute the agreement between Qwen3.5-Plus word-level labels and the mapped EasyTurn ground-truth labels. Then recompute Table 1 using only utterances whose mapped EasyTurn labels are unambiguous, applying the published mapping identically to X2-Turn and SoulX-Duplug. If the accuracy gap over SoulX-Duplug vanishes or reverses under a principled mapping, the central claim is not supported; if the gap persists with high label agreement, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is the evaluation in Table 1. Section 2.3 introduces a five-state taxonomy (<|idle|>, <|noidle|>, <|incomplete|>, <|complete|>, <|backchannel|>) and Section 3.1 generates word-level training labels with Qwen3.5-Plus. Section 4.1 then reports ACC_comp/incomp/bc on EasyTurn-zh/en, comparing 'the last non-idle predicted state against the ground-truth utterance.' The paper never specifies how EasyTurn's native labels (which reference [8] describes as four turn states) are mapped into these five states, nor how the utterance-level ground truth is formed from word-level labels. If the LLM's notion of 'complete' or 'incomplete' differs systematically from EasyTurn's labels, the model is trained on one construct and scored on another. The reported margins over SoulX-Duplug (91.00 vs 77.67 on Chinese ACC_comp; 92.10 vs 89.33 on English ACC_comp) could then reflect the mapping choice rather than genuine turn-state quality. This is the single most load-bearing issue because the central claim is accuracy superiority: if the mapping is inconsistent or arbitrary, the headline numbers do not measure the intended construct.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes X2-Turn, a dual-head streaming model that extends the pretrained Voxtral Realtime delayed-stream ASR backbone with a parallel turn-state head. Both heads operate on shared causal decoder hidden states; word-level turn labels are generated by an LLM and projected onto the 80 ms frame positions of the ASR word-boundary tokens. Training uses two stages: a large-scale Chinese-English ASR adaptation stage and a joint ASR/turn-state fine-tuning stage on EasyTurn and Fisher subsets. Evaluation on the EasyTurn Chinese and English test sets reports turn-state accuracy and latency against cascaded and streaming baselines, together with ASR accuracy on standard benchmarks across different streaming delays. The central claim is that frame-synchronous turn-state prediction with ASR-anchored supervision yields accurate low-latency turn-taking without an auxiliary ASR model.","tokens_in":8806,"tokens_out":9308,"duration_ms":71980,"significance":"The architectural idea is timely and practical: integrating turn-state prediction into a streaming ASR decoder at frame level, with no auxiliary ASR at inference, addresses a real deployment bottleneck for spoken dialogue systems. The ASR-anchored supervision scheme is a clean way to align semantic turn labels with the audio frame timeline, and the controllable streaming delay τ provides a useful accuracy-latency trade-off. Strengths of the paper include the two-stage full fine-tuning on public corpora, bilingual experiments, the τ ablation, and the comparison against several recent baselines. If the evaluation is valid, the reported gains over SoulX-Duplug on streaming turn-state accuracy are a meaningful contribution. However, the quantitative claims are currently not fully substantiated because the mapping from EasyTurn's native labels to the paper's five-state taxonomy is unspecified, and because the ASR comparison table contains an unclear column structure that weakens the related claim.","major_comments":[{"comment":"The paper never specifies how the EasyTurn test set's native turn-state labels are mapped to the proposed five-state taxonomy (<|idle|>, <|noidle|>, <|incomplete|>, <|complete|>, <|backchannel|>), nor how an utterance-level ground-truth label is formed for the 'last non-idle predicted state' comparison. EasyTurn is described in [8] as using four turn states, while the model is trained with LLM-generated word-level labels under a five-state scheme. Without an explicit, consistent mapping, the accuracy numbers in Table 1 may reflect the mapping choice rather than genuine turn-state quality. This is the primary load-bearing issue because the paper's central claim is accuracy superiority over SoulX-Duplug.","section":"§2.3, §3.1, §4.1 (Table 1)"},{"comment":"The claim that Stage2-Turn 'still matches or exceeds Freeze-Omni' is not supported by the table as printed. Table 3 lists four method columns (Uni-ASR, Freeze-Omni, Stage1-ASR, Stage2-Turn) but the rows contain five values, and the delay-header line offers only three delay labels. If the columns are read in the order given, the AISHELL-1 row shows a Stage2-Turn value of 3.94 against a Freeze-Omni value of 2.79, and the GigaSpeech row shows 12.55 against 12.23, both contradicting the claim. The column structure and the claim need to be corrected or qualified.","section":"§4.3 (Table 3)"},{"comment":"No error bars, confidence intervals, or significance tests are reported for the central accuracy comparisons. Some of the margins over SoulX-Duplug are modest, e.g., English ACC_comp is 92.10 vs 89.33 (a 2.77-point gap), and the number of test utterances per split is not given. Without this information, it is difficult to assess whether the reported differences are reliable, especially for the smaller English ACC_incomp gap of 5.27 points and the Chinese ACC_comp gap, which is the core evidence for the method's superiority.","section":"§4.1 (Table 1)"}],"minor_comments":[{"comment":"The LLM annotation procedure is described only through the phrase 'following the annotation criteria defined in Section 2.3', but Section 2.3 contains no prompt, no operational definition of 'complete' vs 'incomplete', and no consistency checks. The exact annotation instructions should be provided for reproducibility.","section":"§3.1"},{"comment":"The caption should define what 'ground-truth utterance' means and how the utterance-level label is obtained from word-level labels. It should also clarify how backchannel predictions are treated on the English test set, which reportedly has no backchannel split.","section":"§4.1 (Table 1)"},{"comment":"Please specify the rounding convention in Eq. (4) (e.g., round-half-up vs floor) and clarify whether s_i is the forced-aligned word onset in seconds; the current notation leaves the units and the mapping to frame indices implicit.","section":"§2.3 (Eq. 4)"},{"comment":"The abbreviation 'V AD' should be written consistently as 'VAD', and 'V oxtral' should be 'Voxtral' in the running text.","section":"§1 and throughout"},{"comment":"The final sentence is incomplete: 'Experiments on the bilingual EasyTurn test sets that X2-Turn achieves...' is missing a verb such as 'show' or 'demonstrate'.","section":"§5 (Conclusion)"}],"recommendation":"major_revision","confidential_remarks":"The main risk is that the evaluation rests on an undocumented label mapping, which is a standard but fixable issue: providing the mapping, the utterance-level aggregation rule, and ideally a small qualitative example would resolve the core concern. The Table 3 column layout should also be corrected before publication. The paper would then be a solid contribution to the streaming turn-taking literature."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"X2-Turn is a sensible extension of Voxtral Realtime: keep the delayed-stream ASR decoder, add a parallel turn state head, and supervise both on the same 80 ms frame grid via ASR-anchored word-level labels. That framing is genuinely new relative to utterance- or chunk-level turn detectors, and the architecture is clean. The latency model (state available at word onset plus tau) is clearly defined, and the ablation over tau shows a predictable accuracy-latency trade-off. Credit where due: this is a real system, trained on a lot of data, and the main streaming comparison to SoulX-Duplug shows a large accuracy gain at comparable latency.\n\nNow the soft spots. The most load-bearing is the evaluation mapping. Section 2.3 defines five states (<|idle|>, <|noidle|>, <|incomplete|>, <|complete|>, <|backchannel|>). Section 4.1 says they compare the last non-idle predicted state against the ground-truth utterance on EasyTurn, but EasyTurn's native labels are four turn states, and the paper never states how those are mapped into the five-state taxonomy. Without that mapping, the reader cannot tell whether ACC_comp measures the intended construct or an artifact of the mapping. This is not a minor omission; it is the central evidence for the paper's main claim. Second, Section 4.3's claim that Stage2-Turn matches or exceeds Freeze-Omni is contradicted by Table 3: on AISHELL-1, GigaSpeech, and LS-other, Stage2-Turn is worse. That sentence needs to be corrected. Third, no error bars, and no released code or data, which makes the strong margins harder to trust. Fourth, the LLM-based word-level labeling with Qwen3.5-Plus is described only briefly; the prompt and any consistency checks are absent, which matters because the same construct is then scored.\n\nNone of these are fatal to the core idea. The architecture is plausible, the latency story is coherent, and the paper is worth engaging. But the missing mapping and the self-contradictory ASR claim prevent me from treating the numbers as established. I'd send it to a serious referee with a request to pin down the mapping, add error bars or per-item analysis, and fix the ASR comparison. The target community (speech, dialogue systems) will want this.","headline":"A useful incremental architecture for frame-synchronous turn state prediction, but the headline accuracy comparison rests on an undocumented label mapping and one ASR claim contradicts its own table.","tokens_in":9324,"tokens_out":2000,"would_cite":false,"duration_ms":17899,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A frame-synchronous dual-head model jointly predicts ASR tokens and turn states every 80 ms, outperforming streaming and cascaded baselines on bilingual turn-taking tests.","keywords":["turn-taking","streaming ASR","frame-synchronous prediction","delayed-stream modeling","dual-head architecture","spoken dialogue systems","backchannel detection","EasyTurn"],"falsifier":"Publish the exact label-to-state mapping for EasyTurn and rescore X2-Turn by comparing each word's final non-idle prediction with the ground truth at word level; if the reported advantage over SoulX-Duplug disappears or drops below a few points under this scoring, the frame-synchronous advantage would be shown to come from utterance-level aggregation rather than from the proposed supervision.","tokens_in":8339,"feed_emoji":"🗣️","tokens_out":7456,"duration_ms":61640,"temperature":0.7,"pith_summary":"This paper claims that turn-taking in spoken dialogue can be predicted frame by frame, at the same 80 ms cadence as streaming speech recognition, by adding a second prediction head to a delayed-stream ASR model. The two heads share the same streaming representations, so a single forward pass emits both transcript tokens and one of five turn states (<|idle|>, <|noidle|>, <|incomplete|>, <|complete|>, <|backchannel|>). Supervision comes from word-level turn labels projected onto the ASR token timeline, which aligns the two tasks at the frame level. On the bilingual EasyTurn test sets, the paper reports that this design beats the streaming baseline SoulX-Duplug on every reported accuracy metric while keeping comparable latency, and it matches or exceeds the best cascaded systems without their VAD segmentation delay. If correct, this removes the need for separate ASR and turn-detection modules in real-time dialogue systems.","feed_headline":"Frame-level turn-taking beats chunk-level streaming baselines","feed_subtitle":"One streaming model emits transcripts and turn states on the same 80 ms clock, beating chunk- and utterance-level baselines","key_machinery":"The central object is the dual-head delayed-stream decoder: a causal audio encoder, a temporal adapter, and a decoder-only language model that emits one token every 80 ms, with ASR and turn-state heads sharing the hidden state. The load-bearing mechanism is ASR-anchored supervision: word-level turn states are projected onto the same 80 ms positions as the word-boundary token [W] and its subword tokens, so both tasks optimize a frame-synchronous discrete timeline. The configurable delay $\\tau$ is conditioned into the decoder through AdaRMSNorm, allowing one model to trade latency against accuracy across multiple operating points.","core_discovery":"X2-Turn extends a pretrained delayed-stream ASR model (Voxtral Realtime) with a parallel turn-state head. At each 80 ms decoding step, the model's single hidden state feeds both the ASR vocabulary head and the turn-state head; the turn-state prediction never feeds back into decoding, so errors cannot corrupt transcription. The turn-state labels are constructed by assigning each word a semantic state with an LLM annotator, then placing the label at the same 80 ms positions as the word's boundary token and subword tokens. The paper's reported results include ACC_comp 91.00, ACC_incomp 93.00, and ACC_bc 96.00 on EasyTurn-zh at 288 ms latency, and ACC_comp 92.10 and ACC_incomp 84.60 on EasyTurn-en at 225 ms, all better than SoulX-Duplug on the corresponding metrics at similar or lower latency. The authors interpret this as showing that frame-synchronous joint modeling can outperform chunk- or utterance-level state prediction while retaining streaming ASR quality.","pith_inferences":["Because the turn head and ASR head share hidden states and predictions are frame-aligned, the same architecture could be extended to jointly predict other frame-level dialogue events such as laughter or speaker change at the same 80 ms cadence; the paper does not test this.","The paper reports utterance-level accuracies, so a per-frame evaluation would sharpen the picture: if frame-level state transitions such as idle-to-incomplete are recovered accurately, the model could directly drive a TTS barge-in policy, an application the paper leaves implicit.","The $\\tau$ sweep suggests a practical deployment recipe: pick $\\tau=320$ ms when latency matters most, since accuracy degrades only mildly, a trade-off a system designer could exploit without retraining. This is our extrapolation from Table 2, not a paper claim."],"forward_implications":["A single streaming model can serve both ASR and turn-state prediction; at inference no external ASR or VAD segmentation is needed, because both heads run on the same forward pass.","On EasyTurn-zh and EasyTurn-en, X2-Turn outperforms SoulX-Duplug on every reported metric (e.g., 91.00 vs 77.67 ACC_comp on Chinese) with similar latency (288 ms vs 295 ms).","Lowering $\\tau$ gives a controlled latency-accuracy trade-off: on Chinese, reducing $\\tau$ from 480 to 320 ms cuts latency from 288 to 120 ms while average accuracy changes from 92.00 to 90.67; on English, it cuts latency from 225 to 65 ms with average accuracy changing from 88.49 to 85.09.","Stage1-ASR at $\\tau=480$ ms beats chunk-based streaming baselines on most test sets (e.g., AISHELL-1 1.48 vs 2.90 and 2.79), so frame-wise delayed-stream ASR remains competitive even before the turn head is added.","Since turn-state predictions are not fed back into decoding, turn-taking errors cannot degrade the ASR transcript."],"supporting_citations":[{"why":"Supplies the pretrained delayed-stream ASR backbone and the 80 ms frame-rate architecture on which the dual heads are built.","marker":"[15]"},{"why":"Defines delayed-stream modeling, the sequence-to-sequence mechanism X2-Turn extends with a parallel turn head.","marker":"[14]"},{"why":"Provides the EasyTurn bilingual test sets and the utterance-level turn-state baselines the reported accuracies are measured against.","marker":"[8]"},{"why":"The streaming chunk-level baseline (SoulX-Duplug) that X2-Turn is directly compared to on turn-state accuracy and latency.","marker":"[11]"},{"why":"Provides word-level timestamps used to anchor word-boundary tokens to word onsets.","marker":"[29]"},{"why":"Supplies the LLM annotator used to assign word-level semantic turn-state labels.","marker":"[16]"},{"why":"Chunk-based streaming ASR baseline used in the Stage1-ASR comparison.","marker":"[30]"},{"why":"Chunk-based streaming full-duplex baseline used in the ASR performance comparison.","marker":"[4]"}],"fun_headline_variants":["Frame-synchronous turn-state head matches ASR at 80ms","Dual-head streaming model predicts turn states frame-level","One model, two heads: ASR plus turn-state at 80ms","Frame-level turn-taking outperforms chunk-level streaming","Turn-state prediction at 80ms beats chunk baselines"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The accuracy claims rest on an unspecified mapping between EasyTurn's original labels and the model's five turn states; if that mapping is looser than the states the model was trained on, the reported accuracies would not measure the intended construct.","fun_headline_variants_meta":{"raw":{"variants":["Frame-synchronous turn-state head matches ASR at 80ms","Dual-head streaming model predicts turn states frame-level","One model, two heads: ASR plus turn-state at 80ms","Frame-level turn-taking outperforms chunk-level streaming","Turn-state prediction at 80ms beats chunk baselines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000174,"raw_usage":{"total_tokens":1275,"prompt_tokens":934,"completion_tokens":341,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":550,"completion_tokens_details":{"reasoning_tokens":257}},"tokens_in":550,"tokens_out":341,"duration_ms":3890,"temperature":1.0,"reasoning_tokens":257,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:07:41.386023+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Publish the exact label-to-state mapping for EasyTurn and rescore X2-Turn by comparing each word's final non-idle prediction with the ground truth at word level; if the reported advantage over SoulX-Duplug disappears or drops below a few points under this scoring, the frame-synchronous advantage would be shown to come from utterance-level aggregation rather than from the proposed supervision.","supporting_citations":[{"cited_title":"Personaplex: V oice and role control for full duplex conversational speech models,","cited_arxiv_id":null,"evidence_quote":"Defines delayed-stream modeling, the sequence-to-sequence mechanism X2-Turn extends with a parallel turn head."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the EasyTurn bilingual test sets and the utterance-level turn-state baselines the reported accuracies are measured against."},{"cited_title":"Easy turn: Integrating acoustic and lin- guistic modalities for robust turn-taking in full-duplex spoken di- alogue systems,","cited_arxiv_id":null,"evidence_quote":"Supplies the LLM annotator used to assign word-level semantic turn-state labels."},{"cited_title":"Wenetspeech: A 10000+ hours multi- domain mandarin corpus for speech recognition,","cited_arxiv_id":null,"evidence_quote":"Chunk-based streaming ASR baseline used in the Stage1-ASR comparison."},{"cited_title":"X2-Turn: Frame-Synchronous Dual-Head Modeling for Joint Streaming ASR and Turn State Prediction","cited_arxiv_id":"2608.10878","evidence_quote":"Chunk-based streaming full-duplex baseline used in the ASR performance comparison."}],"review_version":1}