{"id":"93c3564d-66eb-4d36-8725-f30b67d1e1da","arxiv_id":"2605.23912","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.5,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A 9B SpeechLM trained on 1.38M hours of English/Korean data plus a full-duplex chat extension trained on 119K hours of time-aligned dialogue outperform same-size audio models on speech tasks and FDB turn-taking metrics.","lead":"KRAFTON releases Raon-Speech, a 9B bilingual English–Korean speech language model, and Raon-SpeechChat, a full-duplex real-time dialogue extension. The work packages a staged training recipe, new Korean speech benchmarks, and open checkpoints that beat several same-size audio models on speech-centric tasks while keeping text QA strong.","discovery_kind":"extension","skeptic_critique":{"model":"grok-4.5","headline":"Internal FDB evaluator differences and synthetic duplex data dominate the full-duplex half of the strongest claim.","rationale":"The reader's weakest_assumption correctly isolates the softest joint in the strongest claim: internal FDB evaluator + synthetic duplex dominance. Architecture, staged training, open-source commitment, and the English/Korean SpeechLM tables (Tables 4–5, App. E–F) are solid for a systems report; author-built Korean benches are a secondary concern but secondary to FDB transfer for the duplex half. No stronger internal inconsistency appears. Concrete public-script re-run settles whether the concern lands; until then CONDITIONAL remains the right verdict, so no adjustment.","tokens_in":27605,"tokens_out":488,"duration_ms":22590,"concrete_test":"Re-evaluate Raon-SpeechChat and the four FDB baselines on official FDB v1.0 (and v1.5) using the public reference scripts and judges only; if Pause TOR, Backchannel TOR/Freq, or User-Interruption TOR shift enough to lose best/second-best rank vs PersonaPlex/MiniCPM-o, the 'clearest strengths on FDB v1.0' half of the claim does not transfer.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim pairs Raon-Speech's speech-centric SOTA profile with Raon-SpeechChat's 'clearest strengths' on FDB v1.0 turn-taking/interruption. That second half rests on Appendix G's internal offline evaluator (1.5s/5-word pause takeover vs public 1.0s/3-word; ASR-refined anchors; GPT-5.2 judge) and on a corpus that is ~89% synthetic (106.33K of 119.54K hours), with backchannels/interruptions placed by rules plus a timing model and barge-in truncation (Appendix D). Table 6's headline TOR/Freq/Judge numbers can therefore be inflated relative to public FDB scripts and to real human duplex timing; the paper does not report a side-by-side public-script re-score or a real-only ablation. English/Korean SpeechLM tables are less affected, but the joint claim as stated is load-bearing on the duplex evaluation transfer.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The manuscript presents Raon-Speech, a 9B-parameter English–Korean SpeechLM built by adapting Qwen3-VL-8B-Instruct with speech understanding and Mimi-codec generation modules, and Raon-SpeechChat, a full-duplex extension with a causal encoder, word-level interleaved user-speech/assistant-text/assistant-speech sequences, and SIL/BOW/BC state tokens. Training uses 1.38M hours for Raon-Speech (alignment → end-to-end pre-training with on-policy KD → SimPO post-training) and 119K hours of time-aligned dialogue for Raon-SpeechChat (causal adaptation → duplex pre-training → two-stage fine-tuning). Across Tables 4–5 and appendices, Raon-Speech is compared to eight similarly sized audio foundation models on ASR, speech generation, SpokenQA, speech understanding, and TextQA (including newly released KVoiceBench, KOpenAudioBench, and KMMAU). Raon-SpeechChat is evaluated on Full-Duplex-Bench v1.0/v1.5/v2.0 (Table 6). The paper claims the strongest speech-centric profile for Raon-Speech while preserving text QA, and clearest strengths for Raon-SpeechChat on FDB v1.0 turn-taking and interruption behaviors, with full open-sourcing of checkpoints, pipelines, and a demo.","tokens_in":27963,"tokens_out":1736,"duration_ms":36978,"significance":"If the empirical claims hold under standard evaluation protocols, this is a substantial systems contribution to bilingual SpeechLMs and full-duplex spoken dialogue under 10B parameters. Strengths that should be credited include: (i) a carefully staged recipe that jointly adds speech understanding/generation while retaining strong MMLU-Pro/MMLU-Redux and Korean text QA; (ii) release of three Korean speech benchmarks tailored to culture and language; (iii) open-sourcing of all checkpoints, training/inference code, and an interactive demo, which is rare at this scale and directly enables reproduction and follow-on work; (iv) explicit architectural choices for duplex control (SIL/BOW/BC, text lookahead, causal streaming encoder) with detailed data and hyperparameter documentation (Tables 2–3, Appendices B–D). The work is of clear interest to the speech–language modeling community even if some duplex evaluation details need tightening.","major_comments":[{"comment":"Table 6 and Appendix G: The abstract and §5.2 claim that Raon-SpeechChat shows its “clearest strengths” on FDB v1.0 turn-taking and interruption-sensitive behaviors. Appendix G states that the offline evaluator differs from the public FDB v1.0/v1.5 reference scripts in load-bearing ways (pause takeover: 1.5 s / 5-word vs public 1.0 s / 3-word; ASR-refined anchors and 0.5 s post-anchor margin; GPT-5.2 vs the public judge models). Without a side-by-side re-score under the public scripts (or an official leaderboard submission), the headline TOR/Freq/Judge numbers cannot be treated as comparable to prior FDB reports. Please either re-evaluate all models under the public scripts and report both, or substantially qualify the FDB v1.0 claim and any ranking language that depends on these metrics.","section":null},{"comment":"§4.2 and Appendix D: Full-duplex training uses 119K hours of which 106.33K (~89%) are synthetic, with backchannels/interruptions/overlap placed by annotated roles, rule-based timing, a backchannel prediction model, and barge-in truncation at random word boundaries. Table 6’s interruption and turn-taking gains may therefore partly reflect training–evaluation alignment to synthetic interaction patterns rather than transfer to real human duplex timing. A real-only (or real-heavy) ablation, or at minimum a held-out real-conversation duplex evaluation with the same metrics, is needed to support the claim of “natural real-time conversation” beyond synthetic FDB slices. If such an ablation is infeasible, the abstract/§5.2 claims should be scoped to the synthetic-heavy training regime and offline FDB protocol actually used.","section":null},{"comment":"Table 4 (Speech Generation rows) and the “strongest overall profile on speech-centric tasks” claim (§1, abstract): Kimi-Audio and Audio Flamingo 3 report “–” for LibriSpeech-c and Seed generation, so the intelligibility SOTA is established only against the subset of baselines that emit speech. Please either obtain generation numbers for those models, restrict the generation comparison explicitly to models with reported WER/UTMOS, or rephrase the overall speech-centric ranking so it does not imply a complete head-to-head on generation for all eight baselines. Relatedly, UTMOS is often second-tier while WER/CER is best; the text already notes this, but the abstract’s unqualified “top-performing” framing should match the more nuanced body (intelligibility vs perceptual naturalness).","section":null}],"minor_comments":[{"comment":"§3.1 knowledge distillation: On-policy self-distillation with a text-conditioned teacher is reasonable, but a short note on whether teacher and student share the same decoding temperature/top-p (and any filtering of degenerate trajectories) would help readers assess the mild circularity risk.","section":null},{"comment":"§5.1 SpokenQA: GPT-5.4 (and GPT-5.2 in Appendix G) as judges limits long-term reproducibility. Please pin model versions/dates and release the exact judge prompts (beyond “VoiceBench prompt / translated”) or report inter-judge agreement on a subset.","section":null},{"comment":"Figure 1 and Figure 5: Zero-to-max normalization per axis can visually inflate small absolute gaps. Consider adding raw-score small multiples or a table of normalized values in the appendix so radar plots are not the only comparative view.","section":null},{"comment":"Table 2 / §3.2: Loss weights on PAD/SIL (0.75/0.5 then SIL 0.25) and 50× CE on BC are important free parameters; a one-paragraph sensitivity note (even qualitative) would strengthen confidence that duplex behavior is not brittle to these choices.","section":null},{"comment":"Appendix E: AlpacaEval appears in both VoiceBench and OpenAudioBench aggregates (and likewise KAlpacaEval). State explicitly whether double-counting affects the suite averages or is only for readability of the two official groupings.","section":null},{"comment":"Typos/clarity: “sumsumsumsum” in Figure 2 caption layout; “Librispeech-c Seed” label stacking in Figure 1; occasional missing spaces in compound names. Also fix “Raon-Speech successfully transforms…” repetition between abstract and §1 for concision.","section":null},{"comment":"§7 Future work mentions vision and agentic extensions; a brief limitation paragraph on latency/throughput of the 9–9.8B stack under true streaming (chunk sizes, RTF) would help practitioners more than the current high-level future-work list.","section":null}],"recommendation":"major_revision","confidential_remarks":"This is a strong industrial technical report with unusual openness (weights, code, demo, new KO benchmarks). I would not reject on novelty grounds: the contribution is systems + bilingual coverage + duplex engineering, not a single new theorem. The main risk for the journal is overstated duplex leaderboard claims under a non-public evaluator and heavily synthetic duplex data; if the authors re-run public FDB scripts and add a real-data check or clear scoping, the paper is close to acceptable. Scope fit depends on whether the venue wants full systems/tech-report style papers versus method-focused work—worth an editorial call, not a scientific reject."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"This is a systems technical report that actually ships something useful: a ~9B English–Korean SpeechLM (Raon-Speech) and a full-duplex extension (Raon-SpeechChat), with weights, pipeline, and demo promised open. The architecture is not a new primitive—Qwen3-VL backbone, AuT then Voxtral causal encoder, Mimi RVQ, RCP from Qwen3-Omni, SimPO, on-policy KD—but the staged recipe, 1.38M-hour curated bilingual mix, SIL/BOW/BC state tokens, text lookahead, and three new Korean suites (KVoiceBench, KOpenAudioBench, KMMAU) are the real product.\n\nWhat it does well is the SpeechLM half. Tables 4–5 and the appendices put it against eight similarly sized models on ASR, generation intelligibility, spoken QA, MMAU/KMMAU, and text QA. Korean gains look broad and clean; English is strongest on spoken QA, speech understanding, and generation WER while holding MMLU-Pro/Redux. Training stages, freezes, loss mix, and data filtering are described carefully enough that another lab could try to reproduce the shape of the result. Open release is the right move for this kind of work.\n\nSoft spots, in proportion: the joint “strongest speech-centric + clearest FDB v1.0 strengths” claim leans on the duplex half more than the English/Korean SpeechLM tables do. Full-duplex data is ~89% synthetic with rule/model-placed backchannels and barge-ins; FDB v1.0/v1.5 scores use an internal offline evaluator (different pause thresholds, ASR-refined anchors, GPT-5.2 judge) that Appendix G admits differs from public scripts. No public-script re-score or real-only ablation. UTMOS naturalness is not a win, variance is missing, and author-built Korean benches need external checks. None of that sinks the SpeechLM contribution; it just means the duplex headline should be read as “competitive under our protocol,” not settled SOTA transfer.\n\nWho it’s for: people building bilingual voice agents, Korean speech eval, or full-duplex training recipes. Math is standard CE/KL/SimPO; citations are appropriate. I would send it to peer review as an open systems paper, ask for public FDB re-runs and clearer duplex ablations, and engage the released models myself. Worth the reading group if anyone is actively training SpeechLMs.","headline":"Solid open 9B bilingual SpeechLM with a real recipe and Korean benchmarks; the duplex half of the claim is softer because of synthetic data and an internal FDB evaluator.","tokens_in":28693,"tokens_out":626,"would_cite":true,"duration_ms":10872,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"A 9B bilingual SpeechLM can match or beat larger peers on English and Korean speech tasks while keeping text QA, then extend to full-duplex turn-taking.","keywords":["speech language model","full-duplex dialogue","English-Korean bilingual","knowledge distillation","preference optimization","turn-taking","SpeechLM","open-source checkpoints"],"falsifier":"Re-run the same checkpoints on the public FDB v1.0/v1.5 reference scripts with their original thresholds and judges, and on live human full-duplex sessions with natural interruptions; if turn-taking and interruption metrics fall below the strongest open baselines, the duplex claim does not transfer.","tokens_in":28488,"feed_emoji":"🎙️","tokens_out":769,"duration_ms":10425,"temperature":0.7,"pith_summary":"This technical report claims that a pre-trained text LLM can be turned into a strong English–Korean speech language model without erasing its text ability, and then extended into a full-duplex conversational system that listens and speaks at the same time. Raon-Speech does this by adding speech understanding and generation modules, training on 1.38 million hours of curated speech and text through alignment, knowledge-distillation pre-training, and preference-based post-training. Across 42 English and Korean benchmarks it reports the strongest overall speech-centric profile among eight similarly sized recent audio foundation models, with especially clear gains on spoken question answering, speech understanding, and generated-speech intelligibility, while still leading text QA suites such as MMLU-Pro. Raon-SpeechChat continues from that base on 119K hours of time-aligned dialogue, using a causal encoder, interleaved user-speech/assistant-text/assistant-speech tokens, and explicit when-to-speak state tokens, and shows its clearest strengths on Full-Duplex-Bench v1.0 turn-taking and interruption handling. The authors also release three new Korean spoken benchmarks and open-source the checkpoints, pipeline, and demo, arguing that practical bilingual real-time spoken interaction is now within reach of open ~9B models.","feed_headline":"9B SpeechLM leads bilingual speech tasks, then talks full-duplex","feed_subtitle":"Alignment, distillation, and interleaved state tokens keep text QA while adding real-time turn-taking","key_machinery":"Three-stage SpeechLM recipe plus full-duplex sequence design: freeze-then-unfreeze alignment of speech modules, CE + on-policy KL distillation, SimPO post-training; then causal streaming encoder, word-aligned interleaved user-speech/assistant-text/assistant-speech tokens, and SIL/BOW/BC state tokens that separate when to speak from what to say, with text lookahead.","core_discovery":"A staged recipe—module alignment, end-to-end pre-training with on-policy knowledge distillation from text-conditioned and backbone teachers, then multi-task preference optimization—transforms a pre-trained LLM into a 9B SpeechLM that both understands and generates English and Korean speech while preserving text QA; continuing with a causal encoder and interleaved full-duplex sequences yields competitive real-time turn-taking and interruption behavior on FDB.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["9B SpeechLM tops bilingual speech tasks then adds full-duplex talk","Staged alignment and distillation turn LLM into leading SpeechLM","Raon-Speech leads English-Korean speech understanding and generation","Continual duplex training yields competitive real-time turn-taking","9B model preserves text QA while mastering bilingual speech and chat"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"That scores from the authors' internal Full-Duplex-Bench evaluator and a mostly synthetic, time-aligned dialogue corpus will still hold up under real human duplex conversation and the official public evaluation scripts.","fun_headline_variants_meta":{"raw":{"variants":["9B SpeechLM tops bilingual speech tasks then adds full-duplex talk","Staged alignment and distillation turn LLM into leading SpeechLM","Raon-Speech leads English-Korean speech understanding and generation","Continual duplex training yields competitive real-time turn-taking","9B model preserves text QA while mastering bilingual speech and chat"]},"model":"grok-4.5","effort":"low","cost_usd":0.004778,"raw_usage":{"total_tokens":1421,"prompt_tokens":889,"num_sources_used":0,"completion_tokens":71,"cost_in_usd_ticks":47780000,"prompt_tokens_details":{"text_tokens":889,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":461,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":889,"tokens_out":71,"duration_ms":6800,"temperature":1.0,"reasoning_tokens":461,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-13T08:21:00.726972+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Re-run the same checkpoints on the public FDB v1.0/v1.5 reference scripts with their original thresholds and judges, and on live human full-duplex sessions with natural interruptions; if turn-taking and interruption metrics fall below the strongest open baselines, the duplex claim does not transfer.","supporting_citations":[],"review_version":1}