{"id":"c61179aa-fb68-4af6-90d5-16bb3eb610c5","arxiv_id":"2505.17417","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Fine-tuning an LLM on text instructions converted to Whisper semantic tokens enables it to understand spoken instructions at inference, bypassing TTS and speech instruction data.","lead":"This paper presents Speechless, a method for training voice assistants on text only, by converting text instructions into semantic speech tokens that mimic Whisper encoder outputs. It lets a fine-tuned LLM understand real spoken instructions at inference, without any recorded speech or TTS in the instruction-tuning stage.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The load-bearing risk is a train/inference token distribution gap: the LLM is tuned on Speechless tokens that decode to ~7% WER, but real Vietnamese speech enters through a quantized Whisper pipeline at ~35% WER (Table 1, CV Vi); no Vietnamese speech-instruction eval tests whether the model…","rationale":"The reader's weakest_assumption is essentially correct, and Table 1 lets the concern be sharpened with numbers already in the paper. The quantized Whisper row is the actual inference-time input, not a pessimistic auxiliary baseline; the Speechless rows are the training-time input. Their large gap on Vietnamese Common Voice is exactly the distribution shift that would break the claimed text-to-speech transfer. I considered the 1B/3B vs 8B model discrepancy and the text-input vs speech-input ASR table as alternative candidates; both are serious reproducibility problems, but neither directly targets the central claim the way the token distribution gap does. A positive result on the proposed Vietnamese speech-instruction test would validate the method; a negative result would mean the low-resource claim is unsupported, though the English VoiceBench results still show the idea is not circular. Since the reader already issued a CONDITIONAL verdict, no change is needed, but the stated condition should explicitly include the Vietnamese speech-instruction evaluation.","tokens_in":9304,"tokens_out":8030,"duration_ms":64181,"concrete_test":"Take the released Stage-3 checkpoint (the one actually used for Table 2) and run a held-out set of Vietnamese spoken instructions through the full pipeline: audio -> Whisper-medium encoder -> RVQ quantizer -> LLM. Build the eval set from, e.g., 200 prompts in the released Viettel/Sailor tokenized instruction data, converted to speech with any TTS (or recorded), plus the same prompts in text. Score answer quality on the spoken condition against (a) the same prompts given as text, and (b) the same prompts tokenized by Speechless. Also report decoded WER of the quantized Whisper tokens on these prompts. If spoken-condition accuracy is substantially below the text/Speechless conditions, the core transfer claim for low-resource languages fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central inference path is audio -> Whisper encoder -> RVQ quantizer -> tokens -> LLM, while Stage 3 instruction tuning used only tokens generated by Speechless from text (Sections 2.3 and 3.1). The central claim therefore requires these two token streams to be interchangeable, especially in the low-resource language. Table 1 contains direct evidence that they are not interchangeable in Vietnamese: on Common Voice VI, the quantized Whisper pipeline (the actual inference input) reaches 34.84% WER with beam search and 36.53% with greedy decoding, whereas Speechless-generated tokens decode to 7.08%/5.90% WER. A five-fold WER gap means the LLM will receive an input token distribution that is substantially noisier and different from its training distribution, with no Vietnamese speech-instruction benchmark showing it copes. VoiceBench, the only spoken-instruction evaluation, is English-only and uses TTS audio, so it cannot certify low-resource transfer. The paper itself concedes in Section 5 that robustness in diverse linguistic contexts needs further exploration. This is a load-bearing, testable gap rather than a missing nicety.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"Speechless proposes a three-stage pipeline for training early-fusion speech-language models without speech instruction data or a TTS system. Stage 1 trains a residual vector quantizer (RVQ) on Whisper encoder features; Stage 2 trains a 1B decoder-only model, also called Speechless, to map text (plus duration tokens) into those semantic tokens; Stage 3 uses that model to convert text instruction datasets into semantic-token sequences and fine-tunes an LLM on them. At inference, audio is encoded by Whisper, quantized, and fed to the LLM. The paper reports ASR-style comparisons in Table 1, VoiceBench results in Table 2, and MMLU/VMLU results in Table 3, and it releases code and a tokenized Vietnamese instruction dataset.","tokens_in":9444,"tokens_out":6285,"duration_ms":47238,"significance":"The core idea is attractive and potentially valuable: if text-to-semantic-token alignment can make an LLM robust to quantized Whisper encoder tokens, then low-resource languages that have ASR corpora but no good TTS could obtain speech-instruction tuning. The paper releases code and a tokenized dataset, and the three-stage design is clearly described. However, the current evidence does not substantiate the headline claims. The Speechless rows in Table 1 are text-to-semantic reconstructions, not speech recognition, and no experiment measures whether the final LLM understands real speech instructions in Vietnamese. The main instruction-following results come from an 8B model whose configuration is not described in the methodology. These gaps are central to the claimed contribution.","major_comments":[{"comment":"The rows labeled 'Speechless' are not automatic speech recognition results: they are produced by decoding semantic tokens that the Speechless model generated from text with the Whisper decoder. The actual inference path for the final assistant is audio -> Whisper encoder -> RVQ -> tokens -> LLM, and Table 1 shows that this path has much higher WER than the Speechless path, e.g., 34.84% versus 7.08% with beam search on Common Voice Vietnamese. Since Stage 3 fine-tuning used Speechless-generated tokens, the model is trained on a substantially cleaner token distribution than it receives from real speech at inference. The paper provides no Vietnamese (or other low-resource) spoken-instruction evaluation showing that the LLM copes with this distribution shift, and Section 5 concedes that robustness in diverse linguistic contexts needs further exploration. This gap is load-bearing for the central claim that the method enables voice assistants for low-resource languages.","section":"Section 4.1, Table 1"},{"comment":"The methodology states that the authors chose LLaMA 3.2 1B Base for most experiments and that the 3B model performed similarly, yet Tables 2 and 3 report results for 'Speechless-llama3.1-8B-instruct'. The paper does not describe how this 8B model was trained, whether it uses the same Stage 1 and Stage 2 components, or why the 1B and 3B models were not evaluated on these benchmarks. The main instruction-following evidence therefore comes from an unstated configuration, making the results difficult to interpret and reproduce.","section":"Section 3.2 vs Tables 2 and 3"},{"comment":"VoiceBench is an English-only benchmark, and most of its spoken inputs are generated by Google TTS rather than real human speech. Therefore Table 2 cannot certify the paper's low-resource claim: it does not test whether the model understands Vietnamese speech instructions, nor whether it handles the noisier token stream produced by the quantized Whisper pipeline on real Vietnamese audio. A minimal addition would be a Vietnamese spoken-questions evaluation with audio passed through the actual Whisper + RVQ pipeline, with either transcripts or human judgments as the metric.","section":"Section 4.2, Tables 2 and 3"},{"comment":"Table 1 mixes three different evaluation objects under the heading 'ASR': direct Whisper zero-shot, the quantized Whisper pipeline, and Speechless text-to-semantic generation. The caption says 'Comparative analysis of model performance for general, noisy, and multilingual ASR', but the Speechless rows do not consume audio at all. This is misleading; the table should be reframed as a semantic-token reconstruction comparison, and the quantized Whisper pipeline should be identified as the relevant speech-recognition baseline for the final end-to-end system.","section":"Table 1 caption and Section 4.1"}],"minor_comments":[{"comment":"There is a typo in 'sematic tokens'; it should be 'semantic tokens'.","section":"Section 2.2"},{"comment":"The sentence 'This is also clear when see that with added noise ...' is grammatically incomplete and should be rephrased.","section":"Section 4.1"},{"comment":"The phrase 'MLS Eng 10k' is ambiguous; please specify whether it refers to 10k hours, 10k utterances, or some other quantity.","section":"Section 3.1"},{"comment":"The sentence 'Nevertheless, Speechlessthe methods described in this paper can in principle be applied to noisy data' contains a typo and should be corrected.","section":"Section 5"},{"comment":"The note that 'Experiments other than ours were performed by the VoiceBench authors' means the baselines were not reproduced locally; this should be stated in the main text so readers can calibrate the comparison.","section":"Table 2 caption"},{"comment":"The phrase 'the first pre-tokenized Vietnamese instruction dataset' is a strong claim; it should be substantiated with a comparison to prior released datasets or softened.","section":"Abstract and Section 1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a preprint with several internal inconsistencies that a careful revision could address. The most serious issue is evidentiary: the central low-resource speech-instruction claim is not directly tested, and the reported VoiceBench results use an 8B model not described in the methodology. I would ask the authors to either add a Vietnamese spoken-instruction evaluation under the actual inference path or substantially weaken the claims in the abstract and introduction. The self-citation of Ichigo is understandable given the shared pipeline, but the reliance on it should be more explicit when describing novelty."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper does something concrete and useful: it trains a text-to-semantic-token LM (Speechless) to synthesize Whisper encoder tokens, so instruction tuning can be done from text alone, bypassing TTS. The authors release training code and a tokenized Vietnamese instruction dataset, which is a real service to the community. The recipe is described in enough detail to reproduce, and the low-resource motivation is genuine.\n\nThe novel part is the specific combination: an autoregressive LM mapping text to RVQ-quantized Whisper semantic tokens, with a duration token to compress repeated codes. Prior alignment work is acknowledged, so the novelty is moderate but real.\n\nWhere it gets soft is evaluation. Table 1 conflates two different things. The Speechless rows are text-to-semantic reconstructions: the model reads text and produces tokens that decode to low WER. The Whisper quantized rows are the actual speech recognition pipeline. Calling both 'ASR' is misleading. More importantly, the gap between those rows is the load-bearing risk. On Common Voice Vietnamese, the quantized Whisper pipeline reaches ~35% WER, while Speechless-generated tokens decode at ~7%. At inference, the LLM sees tokens from real speech through that degraded pipeline, but it was trained on clean Speechless tokens. That is a five-fold distribution shift, and the paper offers no Vietnamese speech-instruction benchmark to show the model copes. VoiceBench is English-only and TTS-generated, so it cannot certify low-resource transfer.\n\nThere is also a configuration mismatch: Section 3.2 says Stage 3 uses LLaMA 3.2 1B/3B, while Tables 2 and 3 report LLaMA-3.1-8B-instruct. That needs to be disclosed and fixed. No error bars, and no TTS-based baselines for Vietnamese, which would be the natural comparison for a method that claims to replace TTS.\n\nThese are real problems but they are fixable. The central idea is not circular, the self-citation to Ichigo is fair, and the paper is transparent about limitations in Section 5. The method is plausible and the release helps reproducibility.\n\nThis paper is for researchers building speech assistants in low-resource languages without TTS. It deserves a serious referee, but the revision must add a Vietnamese speech-instruction evaluation, report the exact model configuration, add TTS baselines, and reframe Table 1 honestly. Send it to peer review with major revision.","headline":"A useful TTS-free recipe for speech instruction tuning, but the paper's own numbers reveal a train/inference token gap that the evaluation never closes.","tokens_in":10115,"tokens_out":1773,"would_cite":true,"duration_ms":14355,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Speechless proposes replacing speech instruction data with semantic tokens generated from text, so an LLM can be fine-tuned on text and still hear speech at inference.","keywords":["speech instruction tuning","low-resource languages","text-to-semantics","semantic speech tokens","Whisper encoder alignment","residual vector quantization","voice assistants","Vietnamese speech models"],"falsifier":"Record a set of Vietnamese spoken questions and commands, pass them through the frozen Whisper encoder into the fine-tuned LLM, and score the answers; if the accuracy is substantially below the same questions given as text, the text-to-semantics alignment has not transferred to the instruction domain.","tokens_in":9018,"feed_emoji":"🎙️","tokens_out":8329,"duration_ms":62992,"temperature":0.7,"pith_summary":"This paper tries to establish that an early-fusion voice assistant can be instruction-tuned using only text instructions, with no spoken instruction recordings and no text-to-speech system. The method converts text into discrete semantic tokens that mimic the output of a frozen speech encoder, so the language model is fine-tuned on tokenized text but sees real speech tokens at inference. If the alignment is faithful, the speech instruction bottleneck for low-resource languages reduces to having an ASR dataset and a text instruction corpus, which are far more common than high-quality TTS.","feed_headline":"Train voice assistants on text alone, no TTS required","feed_subtitle":"Synthetic Whisper-compatible speech tokens could let low-resource languages skip speech instruction data entirely.","key_machinery":"The load-bearing object is Speechless, a one-billion-parameter decoder-only model that translates text into the discrete token language of a quantized Whisper encoder. A residual vector quantizer with an enlarged codebook converts Whisper's continuous encoder embeddings into semantic tokens, stripping away acoustic detail while keeping meaning; Speechless is trained on paired ASR transcripts and those tokens, with a duration token compressing repeated codebook entries. This creates an offline 'text-to-semantics' translation step, so instruction tuning of the LLM happens entirely in token space and the frozen Whisper encoder supplies speaker invariance and noise robustness at inference.","core_discovery":"The central claim is that synthetic speech tokens can be produced from text alone and used in place of real speech for instruction tuning. The authors build a residual vector quantizer over the Whisper encoder's representations, then train Speechless, a decoder-only text-to-semantics model, to map ASR transcripts to those quantized tokens. An LLM is then fine-tuned on instruction text whose user turns have been converted offline by Speechless into Whisper-style semantic tokens. At inference, real audio is passed through the same frozen Whisper encoder, and the LLM accepts the resulting tokens as if they had come from the synthetic training data. The paper reports competitive ASR decoding from the generated tokens and English spoken-instruction scores on par with a model trained on 200k real spoken instructions.","pith_inferences":["The paper leaves the speech-to-text alignment transfer unverified in Vietnamese: VoiceBench is English-only and the final model's Vietnamese speech understanding is never measured, so a direct Vietnamese spoken-command evaluation is the natural next experiment.","Because the quantizer and Speechless are trained on clean read speech only, the authors' own results suggest that noisy real-world instructions are a likely failure mode until noisy ASR data is added to Stage 1.","If the alignment holds, the method should extend to any language Whisper already recognizes, since the only language-specific resource needed is ASR transcripts—a claim that a multilingual spoken-instruction test could confirm or refute.","A cheap internal check would be to compare, for the same Vietnamese text, the LLM's answers when given Speechless-generated tokens versus tokens from real recorded speech; high agreement would isolate alignment quality from downstream LLM quality."],"forward_implications":["Any language with an ASR dataset and a text instruction corpus can get a speech-instruction-tuned assistant without recording or synthesizing spoken commands.","TTS quality and speaker diversity stop being constraints on synthetic speech instruction data, since no waveform is ever produced.","Because the speech encoder stays frozen and training happens at token level, the fine-tuning cost is close to text-only instruction tuning rather than speech model training.","The observed drop on text benchmarks after speech instruction tuning indicates a modality-sharing cost that future work will need to manage, and the method inherits Whisper's weaknesses on noisy input."],"supporting_citations":[{"why":"The frozen Whisper encoder defines the semantic token space that both the quantizer and the inference-time speech input share.","marker":"[17]"},{"why":"The preceding voice assistant work supplies the semantic token handling and the English instruction data used in fine-tuning.","marker":"[1]"},{"why":"The Vietnamese ASR speech-text corpus provides the paired data for training the quantizer and the text-to-semantics model.","marker":"[23]"},{"why":"The English read-speech corpus supplies the English side of the Stage 1 and Stage 2 training data.","marker":"[22]"},{"why":"The clean English speech corpus contributes additional text-to-semantic training pairs in Stage 2.","marker":"[24]"},{"why":"The large English text corpus gives the LLM pretraining material after conversion into synthetic semantic tokens.","marker":"[25]"},{"why":"The instruction-tuning recipe this work adapts, with the user turn replaced by semantic tokens.","marker":"[21]"},{"why":"The spoken-instruction benchmark provides the English evaluation that supports the transfer claim.","marker":"[28]"},{"why":"A reference system trained on 200k real spoken instructions serves as the main comparison point for the benchmark results.","marker":"[4]"}],"fun_headline_variants":["No TTS needed: train voice assistants on text alone","Synthetic Whisper tokens replace speech data for instruction tuning","Text-to-speech-free training for low-resource voice assistants","Train on text, hear at inference: TTS-free speech understanding","Whisper-aligned semantics let models learn speech from text"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method stands or falls on whether the sequences of semantic tokens generated from text are close enough to the sequences Whisper produces from real speech, for questions and commands never seen during training.","fun_headline_variants_meta":{"raw":{"variants":["No TTS needed: train voice assistants on text alone","Synthetic Whisper tokens replace speech data for instruction tuning","Text-to-speech-free training for low-resource voice assistants","Train on text, hear at inference: TTS-free speech understanding","Whisper-aligned semantics let models learn speech from text"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000208,"raw_usage":{"total_tokens":1348,"prompt_tokens":835,"completion_tokens":513,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":451,"completion_tokens_details":{"reasoning_tokens":428}},"tokens_in":451,"tokens_out":513,"duration_ms":4162,"temperature":1.0,"reasoning_tokens":428,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:47:30.681059+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Record a set of Vietnamese spoken questions and commands, pass them through the frozen Whisper encoder into the fine-tuned LLM, and score the answers; if the accuracy is substantially below the same questions given as text, the text-to-semantics alignment has not transferred to the instruction domain.","supporting_citations":[{"cited_title":"An efficient and high fidelity vietnamese streaming end-to-end speech synthesis,","cited_arxiv_id":null,"evidence_quote":"The frozen Whisper encoder defines the semantic token space that both the quantizer and the inference-time speech input share."},{"cited_title":"Astra: Aligning speech and text representa- tions for asr without sampling,","cited_arxiv_id":null,"evidence_quote":"The Vietnamese ASR speech-text corpus provides the paired data for training the quantizer and the text-to-semantics model."},{"cited_title":"An analysis of semantically-aligned speech-text embeddings,","cited_arxiv_id":null,"evidence_quote":"The English read-speech corpus supplies the English side of the Stage 1 and Stage 2 training data."},{"cited_title":"Speecht5: Unified-modal encoder-decoder pre-training for spoken language processing,","cited_arxiv_id":null,"evidence_quote":"The large English text corpus gives the LLM pretraining material after conversion into synthetic semantic tokens."},{"cited_title":"Alpaca: A strong, replicable instruction-following model,","cited_arxiv_id":null,"evidence_quote":"The instruction-tuning recipe this work adapts, with the user turn replaced by semantic tokens."},{"cited_title":"ASR and Speechless Comparisons To evaluate the performance of the Speechless model alone, we make use of ASR test sets","cited_arxiv_id":null,"evidence_quote":"A reference system trained on 200k real spoken instructions serves as the main comparison point for the benchmark results."}],"review_version":1}