{"id":"4932ad6d-7672-4080-96e8-4bad43c60a3e","arxiv_id":"2506.17611","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A new open family of speech-language models trained on public data matches or beats prior systems across ASR, TTS, and text benchmarks.","lead":"This paper presents OpusLM, a family of open speech-language models from 135M to 7B parameters that handle speech recognition, speech synthesis, and text together. The models are trained entirely on public data, and the project releases code, checkpoints, training logs, and a data recipe.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"TTS ASR-WER comparison in Table 1 may conflate evaluation protocols: OpusLM reports VERSA-based ASR-WER while ChatTTS/CosyVoice/Parler-TTS rows likely cite their own published numbers, making the claimed TTS superiority over dedicated TTS systems unverified.","rationale":"The reader's weakest_assumption (cross-system comparison relies on each system's own evaluation setup) is correct and important, and I agree with CONDITIONAL. My partial agreement is because the reader framed the concern mainly around ASR margins (2.3 vs 2.8), whereas the comparability problem is sharper in the TTS column: the paper compares a VERSA-measured TTS ASR-WER against TTS systems' self-reported numbers, and the prose draws a direct superiority conclusion ('outperforms all other competitors') from this. A single matched re-evaluation of the most accessible baseline (Parler-TTS) would settle it. I do not find a more central flaw: the internal scaling, loss-region, batch-size, in-domain-text, and annealing experiments are self-consistent, the 7B unconverged checkpoint is disclosed, the proxies (ASR-WER, SPK-SIM, proxy MOS) are standard in this field, and public data/release strengthens credibility. The concern is therefore about the strength of the external comparison, not the internal validity; the conditional verdict stands, contingent on a matched TTS harness.","tokens_in":9903,"tokens_out":1534,"duration_ms":15544,"concrete_test":"Re-run the TTS comparison in a matched harness: for each of ChatTTS, CosyVoice, and Parler-TTS (or at least Parler-TTS, whose weights are open), synthesize the same LibriSpeech/LibriTTS prompt-target test subset with the published recipe, decode with the same VERSA ASR model and text normalization used for OpusLM, and compare ASR-WER. If Parler-TTS or CosyVoice matches or beats OpusLM's 4.0% under this matched protocol, the claimed TTS superiority is not established.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The headline claim that OpusLMs achieve 'comparable or superior' performance with existing SpeechLMs rests on Table 1's cross-system numbers. The reader flagged ASR comparability; the more specific issue is the TTS column. OpusLM's TTS ASR-WER (4.0-4.6) is measured with VERSA using whisper-large-v3, while ChatTTS (7.1), CosyVoice (5.0), and Parler-TTS (4.7) are almost certainly their paper-reported numbers under their own recognizers, test sets, and normalization. The paper provides no evidence these are interchangeable: no re-decoding of prior systems in VERSA, no comparison of recognizer choice, no matched text normalization. The 4.0% best TTS WER is presented as outperforming all competitors, but TTS ASR-WER is notoriously sensitive to the ASR model and normalization. Additionally, the paper's own footnote 15 concedes OpusLM-7B TTS is an intermediate checkpoint with high WER, so the headline TTS result rests entirely on the 1.7B model with one Opt-B annealing choice. This is a load-bearing comparability premise for the strong TTS claim, separate from the internal scaling and annealing results, which are self-consistent and well-supported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes OpusLM, a family of decoder-only speech language models (135M, 360M, 1.7B, and 7B) initialized from pretrained text LLMs and continuously pretrained on 213K hours of public English speech-text pairs and 292B text-only tokens. Audio is represented by semantic plus acoustic discrete tokens in a delay-interleaved multi-stream architecture, with a text:semantic:acoustic loss reweighting. Training proceeds in a pretraining stage followed by a short annealing stage with two data compositions. The authors report LibriSpeech Test-Clean ASR WER of 2.3% (7B), TTS ASR-WER of 4.0% (1.7B), and MMLU of 59.0 (7B) with only 2.7 points of text degradation from the base LLM. They also present scaling experiments (135M/360M/1.7B), design-choice ablations (loss region, batch size, text corpus domain), and annealing experiments showing in-domain and partially out-of-domain gains. All code, data, checkpoints, and training logs are released.","tokens_in":10326,"tokens_out":4925,"duration_ms":55868,"significance":"If the headline results hold under comparable evaluation, the paper would be a valuable demonstration that open, public-data SpeechLMs can approach or match much larger or proprietary systems, and the full release of code, data, checkpoints, and logs is a genuine contribution to reproducibility in this area. The internal scaling and annealing ablations are self-consistent and provide useful practical guidance for SpeechLM pretraining. The architecture description is clear and the use of standard, publicly available tokenizers and LLM initializations lowers the barrier for follow-up work. However, the paper's central comparative claim—'comparable or even superior' performance—rests on cross-system numbers whose evaluation protocols are not matched, and the most prominent 7B results come from an explicitly unconverged checkpoint. These issues need to be addressed before the comparative claims can be accepted as stated.","major_comments":[{"comment":"The TTS comparison in Table 1 is not based on a matched evaluation protocol. The paper states in Section 3.1 that its own TTS evaluation uses VERSA with ASR-WER, speaker similarity, and proxy MOS, but it does not state that the ChatTTS, CosyVoice, and Parler-TTS rows (Table 1, rows 3-5) were re-evaluated under the same VERSA pipeline. Those numbers are almost certainly taken from each system's own publications, which involve different recognizers, test sets, and text normalization. Because TTS ASR-WER is highly sensitive to the ASR model and normalization, the claimed 4.0% versus 4.7% margin over Parler-TTS cannot be taken as evidence of superiority. Please either re-run the comparison systems under the identical VERSA evaluation or explicitly label Table 1 as a collection of numbers from heterogeneous protocols and soften the 'outperforms all other competitors' claim accordingly.","section":"Section 3.1 and Table 1"},{"comment":"The 7B model is an intermediate, unconverged checkpoint. Footnote 9 says OpusLM-7B was trained for only 250k updates and 'is not fully converged,' and footnote 15 says its TTS is weaker because 'TTS converges slower than ASR.' Despite this, the abstract and Table 1 present the 7B ASR WER of 2.3% and MMLU of 59.0 as headline results, and Section 3.2 states the 7B model 'outperforms all other SpeechLMs on the MMLU metric.' The main text should clearly mark these as intermediate-checkpoint numbers and either defer comparative claims for 7B or report the final converged numbers. As written, the paper gives the impression that OpusLM-7B is a finished model, which conflicts with its own appended limitation.","section":"Section 2.3 footnote 9, Table 1, Section 3.2"},{"comment":"The claim that annealing improves performance 'in general rather than adapting to the annealing data domain' is not fully supported. For Opt-B, out-of-domain ASR WER worsens from 13.9% to 14.5% (Table 3), and the only clear out-of-domain wins are in TTS. Moreover, the in-domain evaluation uses LibriSpeech test sets while Opt-A contains LibriSpeech and Opt-B contains LibriTTS (which the paper itself, in footnote 14, calls a re-segmented version of LibriSpeech), so the in-domain gains are at least partly explained by direct overlap with the annealing data. The paper should separate the evidence for ASR and TTS, report an out-of-domain ASR setting where annealing helps, or explicitly hedge the generalization claim to the TTS task.","section":"Section 3.4 and Table 3"}],"minor_comments":[{"comment":"Typo: 'two trails of annealing' should be 'two trials of annealing.'","section":"Section 2.3"},{"comment":"'using lm eval' should be written as 'using lm-evaluation-harness' with a proper citation, since the current text is ambiguous about the exact harness version and settings.","section":"Section 3.1"},{"comment":"The word 'parallelly' is nonstandard; use 'in parallel' or 'simultaneously.'","section":"Section 2.2, Eq. (4)"},{"comment":"The sentence 'the 4M batch size is translated to 125k updates to the model' is unclear; please clarify whether this refers to the number of optimizer steps or the effective number of epochs, and how it relates to the comparison at 30k and 60k updates.","section":"Table 2.b"},{"comment":"The abstract contains a spacing error ('presentsOpenUnifiedSpeechLanguage Models'), and the Figure 2 caption has a stray space in 'T ext MMLU'; these should be corrected.","section":"Abstract and Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The paper is honest about limitations in footnotes (unconverged 7B, in-domain annealing overlap, longer-context issues), but those footnotes are in tension with the abstract and Section 3.2's comparative claims. The internal scaling and annealing experiments are solid and the open-release artifacts are commendable. I believe the manuscript is salvageable with a major revision that either matches the evaluation protocols for Table 1 or clearly reframes the claims as protocol-dependent. I do not think rejection is warranted, because the core methodological contributions and the 1.7B internal results appear sound."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know about OpusLM: it's a genuinely useful open SpeechLM family with released checkpoints, data recipes, and training logs. The most durable contributions are the scaling finding (speech-text joint pre-training does not work below roughly 1.7B) and the annealing sensitivity result (same checkpoint, two data compositions, noticeably different out-of-domain TTS WER). The paper also shows that keeping the text corpus in-domain with the base LLM matters a lot for preserving MMLU. Those are empirical results people will cite.\n\nWhat the paper doesn't nail is the headline comparison against prior systems. Table 1 mixes numbers evaluated under different protocols. OpusLM's TTS WER is measured with VERSA, while ChatTTS, CosyVoice, and Parler-TTS rows are almost certainly the numbers from their own papers. Nobody re-decoded those systems in the same harness. The 4.0% versus 4.7% gap over Parler-TTS is small enough that recognizer choice and normalization could flip it. The same issue exists in the ASR column. This is load-bearing only for the \"comparable or superior\" claim; the scaling and annealing findings don't depend on it.\n\nThe paper is honest about its own limitations: the 7B model is an intermediate checkpoint, TTS quality uses proxy metrics, and long-form TTS results are post-selected. But it doesn't give seeds or error bars, so the annealing and scaling conclusions rest on single runs. For a 213K-hour training study that's understandable, but some margins should be read with caution.\n\nThe stress-test note about TTS evaluation protocols is correct. The authors should either re-evaluate key baselines in VERSA or soften the claim. That's a revision request, not a rejection.\n\nI'd send this to review. It's a substantial open-system contribution from a credible group, and the internal results are coherent. A good referee would push on evaluation comparability and perhaps ask for multi-seed calibration on smaller models. Anyone working on open speech-language models—especially on data curation and training schedules—will get value from this paper.","headline":"Solid open SpeechLM family with valuable scaling and annealing findings; headline cross-system comparisons need a common harness.","tokens_in":10794,"tokens_out":2210,"would_cite":true,"duration_ms":24278,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A fully open family of speech language models, trained on public data alone and initialized from text LLMs, matches or beats prior systems on speech recognition, speech synthesis, and text-only benchmarks.","keywords":["speech language model","speech foundation model","automatic speech recognition","text-to-speech","delay interleave architecture","multi-stream tokenization","open-source model release","model scaling"],"falsifier":"Run the Table 1 baselines, including the strongest single-task ASR and TTS models and the multi-task speech LMs, through the exact OpusLM inference protocol — greedy decoding for ASR, top-30 sampling at temperature 0.7 for TTS, the same TTS scoring harness, no speaker-prompt or sample selection, and identical text normalization — and check whether the OpusLMs' 2.3% Test-Clean WER, 4.0% TTS WER, and 59.0 MMLU still beat them; if re-measured baselines close the gap, the 'comparable or superior' claim would not survive.","tokens_in":9697,"feed_emoji":"🎙️","tokens_out":15563,"duration_ms":152586,"temperature":0.7,"pith_summary":"The paper claims that a competitive speech language model can be built entirely from public materials. Its OpusLMs, initialized from pretrained text LLMs and pre-trained on 213K hours of open English speech-text pairs plus 292B text-only tokens, reach a 2.3% word error rate on LibriSpeech Test-Clean, a 4.0% WER in text-to-speech, and an MMLU score of 59.0 on the 7B model, while losing only 2.7 points of text capability relative to the base LLM. The paper's case is that the entire recipe — architecture, data, and training schedule — for a speech-text model that rivals larger or proprietary-data systems is now reproducible and auditable rather than hidden inside closed training pipelines. It also argues that the joint speech-text setup needs at least about 1.7B parameters to work, and that annealing on curated data helps but is sensitive to which data is chosen.","feed_headline":"2.3% error: public-data speech model matches bigger rivals","feed_subtitle":"A fully open 7B model handles recognition, synthesis, and text, with text quality down just 2.7 points.","key_machinery":"The load-bearing mechanism is the delay interleave architecture operating on a multi-stream token grid. Each frame's $N=9$ tokens — $x_{t,1}$ semantic, $x_{t,2},\\dots,x_{t,9}$ acoustic — are delayed by $n-1$ frames, the delayed embeddings are summed frame-wise, and one causal Transformer predicts all tokens of the next frame in parallel with trainable level biases $b_n$, which is what keeps the inference cost at $O(T)$ independent of the number of streams. Two further pieces carry the results: the joint vocabulary whose padding token has zero embedding, so that text-only sequences behave identically to the source text LLM and the pretrained text knowledge is inherited; and the two-stage schedule in which a long pre-training phase with 1M-frame batches, an in-domain text corpus, and target-region loss is followed by annealing with curated data and spliced long-form speech, which the paper shows turns one-to-two-minute recognition and synthesis from infeasible into usable.","core_discovery":"OpusLMs are decoder-only Transformers that read and write a multi-stream token grid rather than a single token string: each 50Hz audio frame carries one semantic token and eight acoustic codec tokens ($N=9$), while text frames are padded to the same width with a zero-embedding pad. The central design, adopted from prior interleaving work, is the delay interleave architecture, in which token $x_{t,n}$ is moved $n-1$ frames later, the delayed embeddings in each frame are summed, and one causal Transformer predicts the next frame's tokens in parallel using per-stream bias vectors; this preserves intra-frame autoregression, keeps inference at $O(T)$ regardless of the stream count, and for text-only input reduces exactly to the behaviour of the source text LLM. Initialized from the SmolLM2 series (135M, 360M, and 1.7B) and from OLMo-2-7B, the models are trained on spliced task sequences for ASR, TTS, speech-only, and text-only with a joint vocabulary and task identifier tokens, then annealed on curated high-quality data spliced to up to two minutes of speech. The paper reports 2.3% WER on LibriSpeech Test-Clean (7B), 4.0% TTS WER (1.7B), and 59.0 MMLU (7B) with a 2.7-point degradation, and uses these to argue that fully open, public-data SpeechLMs are comparable or superior to prior systems. Its stated contribution is the articulation of the whole system design — tokenization, multi-stream modeling, and multi-stage training — rather than any single new component.","pith_inferences":["One testable extension: if the pretrained text backbone carries much of the gain, initializing from a weaker public base LLM with the same speech data should shrink the ASR and TTS advantages proportionally.","Since inference cost is $O(T)$ regardless of stream count $N$, a wider codec with more acoustic tokens per frame could be plugged in at the same inference cost, offering a direct test of whether synthesis quality improves with stream width — a knob this paper does not turn.","The two annealing trials differ in data composition but not in schedule, so the reported gains leave open whether the decay itself or the better data produces them; rerunning annealing with the compositions swapped would separate the two effects.","The 7B model is an intermediate checkpoint stopped halfway through its scheduled updates by the paper's own account, so the published 7B numbers are plausibly a floor, and the paper's scaling story implies full convergence would widen the gap over the 1.7B model."],"forward_implications":["Future SpeechLM work can start from fully released weights, data, code, and training logs, because the paper shows public resources are enough to reach the reported performance.","One model can serve ASR, TTS, and text tasks with text skill largely intact, so applications do not need separate speech and text stacks.","The 135M and 360M variants collapse on all three tasks in this joint setup, making scale past roughly 1.7B a precondition for the unified paradigm rather than an optimization detail.","Annealing with curated data improves out-of-domain performance and makes long-form speech tractable, so the annealing practice from text LLM development carries over to speech pre-training.","Because text-only input collapses to the base text LLM's behavior, the trained models can be dropped into existing text LLM inference pipelines as replacements."],"supporting_citations":[{"why":"Supplies the delay interleave multi-stream architecture, the paper's central mechanism for modeling interleaved speech and text tokens.","marker":"[40]"},{"why":"Provides the 7B base model, its transparent text corpus that keeps the text stream in-domain, and the annealing practice the training schedule copies.","marker":"[23]"},{"why":"Provides the SmolLM2 base models and corpus composition used for the 135M, 360M, and 1.7B variants and for the in-domain text finding.","marker":"[36]"},{"why":"Supplies the open SpeechLM toolkit whose modular features, public tokenizers, and recipes the whole implementation is built on.","marker":"[25]"},{"why":"Provides the YODAS public speech corpus making up the main volume of the 213K-hour training mixture.","marker":"[41]"},{"why":"Provides the processing pipeline that produced the YODAS subset, supporting the public-data sufficiency claim.","marker":"[42]"},{"why":"Anchors the headline ASR measurements through the LibriSpeech Test-Clean and Test-Other subsets.","marker":"[26]"},{"why":"Is the MMLU benchmark used to claim text capability is preserved with only a 2.7-point drop.","marker":"[27]"},{"why":"Is the strongest single-task ASR baseline the results must match and the recognizer used for TTS word-error scoring.","marker":"[32]"},{"why":"Supplies the TTS evaluation harness, so the synthesis numbers are scored inside the paper's own pipeline.","marker":"[48]"}],"fun_headline_variants":["Open speech LM rivals bigger models at 2.3% WER","Fully open 7B speech LM matches bigger rivals","Open SpeechLM: ASR, TTS, and text in one 7B model","Public-data speech LM: 2.3% WER, 4.0% TTS WER","Open 7B model does speech recognition, synthesis, and text"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the headline comparisons are fair: the baseline numbers are the ones each prior system reported under its own evaluation setup, and OpusLM was not re-measured against them in one shared harness, so the claimed margins could narrow if every system were scored identically.","fun_headline_variants_meta":{"raw":{"variants":["Open speech LM rivals bigger models at 2.3% WER","Fully open 7B speech LM matches bigger rivals","Open SpeechLM: ASR, TTS, and text in one 7B model","Public-data speech LM: 2.3% WER, 4.0% TTS WER","Open 7B model does speech recognition, synthesis, and text"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000479,"raw_usage":{"total_tokens":2425,"prompt_tokens":1055,"completion_tokens":1370,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":671,"completion_tokens_details":{"reasoning_tokens":1266}},"tokens_in":671,"tokens_out":1370,"duration_ms":9650,"temperature":1.0,"reasoning_tokens":1266,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:30:20.121987+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the Table 1 baselines, including the strongest single-task ASR and TTS models and the multi-task speech LMs, through the exact OpusLM inference protocol — greedy decoding for ASR, top-30 sampling at temperature 0.7 for TTS, the same TTS scoring harness, no speaker-prompt or sample selection, and identical text normalization — and check whether the OpusLMs' 2.3% Test-Clean WER, 4.0% TTS WER, and 59.0 MMLU still beat them; if re-measured baselines close the gap, the 'comparable or superior' claim would not survive.","supporting_citations":[{"cited_title":"Role of intelligence tests in speech/language,","cited_arxiv_id":null,"evidence_quote":"Provides the 7B base model, its transparent text corpus that keeps the text stream in-domain, and the annealing practice the training schedule copies."},{"cited_title":"Soundstream: An end-to-end neural audio codec,","cited_arxiv_id":null,"evidence_quote":"Provides the SmolLM2 base models and corpus composition used for the 135M, 360M, and 1.7B variants and for the in-domain text finding."},{"cited_title":"Owsm v3.1: Better and faster open whisper-style speech models based on e-branchformer,","cited_arxiv_id":null,"evidence_quote":"Anchors the headline ASR measurements through the LibriSpeech Test-Clean and Test-Other subsets."},{"cited_title":"On the effects of heterogeneous data sources on speech-to-text foundation models,","cited_arxiv_id":null,"evidence_quote":"Is the MMLU benchmark used to claim text capability is preserved with only a 2.7-point drop."},{"cited_title":"Zero: Memory optimizations toward train- ing trillion parameter models,","cited_arxiv_id":null,"evidence_quote":"Supplies the TTS evaluation harness, so the synthesis numbers are scored inside the paper's own pipeline."}],"review_version":1}