{"id":"b17e38cf-f869-4f0c-b1b1-d862ef123682","arxiv_id":"2509.09174","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"EchoX trains a speech decoder on pseudo-labels generated by a frozen text-to-codec module, reducing the acoustic-semantic gap and preserving reasoning in speech-to-speech LLMs with far less training data.","lead":"EchoX is a training method for speech-to-speech language models that uses a frozen text-to-speech module to generate pseudo speech targets from the model's hidden states. The authors report that EchoX, trained on about 6,000 hours of audio, matches or beats several larger models trained on far more data on spoken QA benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 4's 12.8-point gain is not over cascade (EchoX is 6.1 points below Cascade); the real comparison is vs the under-specified 'w/o Echo' baseline, and the pseudo-labels come from a non-stationary greedy text loop.","rationale":"I read the paper in good faith: EchoX is a reasonable three-stage framework, and Table 4 does provide a controlled comparison showing a 12.8-point gain from adding Echo training. That is the strongest evidence for the central claim. However, the reader's strongest_claim overstates this as a gain over the cascade baseline, which is not what Table 4 shows. The genuine load-bearing uncertainty is whether the gain is attributable to the proposed acoustic-semantic alignment mechanism or to details of the echo-target construction and the under-described baseline. The pseudo-labels Y' come from a frozen T2C evaluated on the model's own greedy decoding X', which is not static during Stage III training. The paper does not specify whether targets are re-generated or cached, nor how sensitive the result is to target noise. This is a concrete, testable weakness rather than a rejection: an oracle-text target experiment would disentangle the contribution of the echo loop from the contribution of having a good speech decoder. Since the reader already assigned CONDITIONAL, my recommendation is no change; the concern strengthens the reasons for the condition but does not shift the verdict.","tokens_in":16379,"tokens_out":8066,"duration_ms":98760,"concrete_test":"Run the Stage-III ablation with echo targets generated from ground-truth text X rather than greedy-decoded X' under otherwise identical settings, and additionally run targets cached once at initialization vs refreshed every 1k steps. Report Table-4 averages over at least 3 seeds. If oracle-text targets do not improve over greedy targets, and cached vs refreshed targets differ by no more than seed noise, then the self-echo target loop is not the bottleneck. If they differ materially, the central claim must be qualified by target noise and non-stationarity.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The reader's '12.8-point average gain over the cascade baseline' misreads Table 4. EchoX is 37.1 vs Cascade 43.2 (6.1 points lower); the 12.8-point difference is EchoX vs 'EchoX w/o Echo training' (24.3). The central causal claim therefore rests on that ablation, not on beating cascade. The ablation's control condition is not fully specified: what replaces the Echo loss? Is the Echo decoder still initialized from T2C? Is the same denoising adapter used? Are targets ground-truth TTS outputs? Initialization or target-supervision differences could explain part of the gain.\n\nThe more load-bearing weakness is the echo-target construction in Sec. 2.4. Y' is produced by a frozen T2C from X', which is the greedy decode of the current Stage-III model. Because the LoRA is being updated, X' (and hence Y') changes during training; the paper does not state whether targets are cached or refreshed. If stale, the decoder is trained against an outdated model's text; if refreshed, targets are non-stationary and training can chase a moving distribution. No experiment controls for the quality or variance of X'. The mechanism claim—that Echo training aligns acoustic and semantic spaces—needs evidence that the improvement is not just the decoder learning to speak the model's own text errors.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"EchoX proposes a three-stage training pipeline for speech-to-speech LLMs. Stage I trains a speech-to-text LLM, Stage II trains a frozen text-to-codec (T2C) module, and Stage III introduces an 'Echo decoder' initialized from the T2C parameters. During Stage III, the S2T LLM's hidden states are greedy-decoded into text, passed through the frozen T2C module to produce pseudo speech-token targets Y', and the Echo decoder is trained to predict Y' from the hidden states, with a denoising adapter and a cosine-similarity loss. The model also uses 'unit language' speech tokens and a streaming generation trigger. Experiments on Llama Questions, Web Questions, and TriviaQA report speech-to-speech averages of 37.1 for EchoX-3B and 46.3 for EchoX-8B, competitive with models trained on much more data. The controlled comparison in Table 4 shows a 12.8-point gain over an under-specified 'EchoX w/o Echo training' condition, but EchoX remains 6.1 points below the Cascade condition.","tokens_in":16776,"tokens_out":6510,"duration_ms":82488,"significance":"If the central effect is real, the paper introduces a practical way to train speech-output LLMs without requiring parallel ground-truth audio for every response: the frozen T2C module acts as a target generator, and the denoising adapter is simple. The reported results with ~6k hours of training data, alongside a released code repository and an online human preference evaluation, make this a potentially useful contribution to the growing SLLM literature. However, the paper's headline claim is not supported by the controlled comparison as presented: the positive ablation is versus a poorly specified 'w/o Echo training' row, not versus the Cascade row. The pseudo-labeling loop also raises non-stationarity and self-distillation concerns that are not addressed. The significance therefore depends on whether the authors can clarify the ablation and provide evidence about target quality and target freshness.","major_comments":[{"comment":"The central comparison is mis-stated. EchoX-3B averages 37.1, which is 6.1 points below the Cascade row (43.2). The 12.8-point difference is between EchoX and 'EchoX w/o Echo training' (24.3), not 'over the cascade baseline' as claimed in the abstract and §5.1. Please specify exactly what the 'EchoX w/o Echo training' condition is: does it use the same Echo decoder and denoising adapter but train on ground-truth TTS codec targets? Is the decoder still initialized from T2C? What replaces L_Echo? The Cascade row is also unspecified: is it the same T2C/vocoder applied to the Stage-I text output? If the control condition uses ground-truth speech targets, then the result that pseudo-labels outperform ground-truth targets is surprising and needs a mechanistic explanation. As written, the load-bearing controlled claim rests on an ablation whose setup is not described.","section":"Table 4 / §5.1"},{"comment":"The pseudo-label Y' is generated by greedy-decoding X' from the current S2T LLM and passing X' through the frozen T2C. Because the LoRA parameters are being updated during Stage III, X' and hence Y' are non-stationary. The paper does not state whether Y' is cached or recomputed each step. If cached, the Echo decoder is trained against a stale model's text; if recomputed, the training target drifts. More fundamentally, Y' is a self-generated target, so the Echo loss is a form of self-distillation: it may teach the decoder to speak the model's own text errors rather than to align with an external semantic ground truth. Please report an ablation on target freshness (e.g., cached vs. refreshed pseudo-labels) and an analysis of X' quality (e.g., WER or QA accuracy of the greedy-decoded text). A comparison against training the Echo decoder with ground-truth ⟨text, codec⟩ targets would directly","section":"§2.4, Eqs. (1)–(2)"},{"comment":"Stage III trains the Echo decoder on the full response hidden sequence H = {h1,...,hn}, but streaming inference feeds only partial hidden subsequences to the Echo decoder, as decided by the cosine-similarity trigger and local-extremum criterion. This train/inference mismatch is not analyzed. Table 6 shows that streaming decoding is not catastrophic, but the paper does not explain why the decoder, trained on full sequences, transfers to truncated inputs. If the streaming trigger is purely heuristic, its interaction with the Echo decoder's autoregressive conditioning should be tested, for example by fine-tuning on randomly segmented hidden states or by ablating the trigger design.","section":"§2.4 vs. §2.6"},{"comment":"The 'acoustic-semantic gap' evidence in Figure 6 is under-specified. The figure reports similarity between 'Hi'/'Hello' and 'Hi'/'High' for the S2T model, the interleaving model, and speech tokens, but no details are given about which hidden layer is used, how word-level representations are pooled, how many utterances are averaged, or whether the differences are stable across runs. As it stands, this is an illustrative correlation, not support for the causal claim that the acoustic-semantic gap is the main cause of knowledge degradation. Please provide the computation details and, ideally, error bars or a larger sample.","section":"§5.2 / Figure 6"}],"minor_comments":[{"comment":"The header 'A vg.' should be 'Avg.' Also, Table 3 omits TriviaQA for LLaMA-Omni2 rows; please indicate whether this is 'not reported' or 'not evaluated'.","section":"Tables 2–4"},{"comment":"Eq. (1) defines L_Echo as a log-probability without a negative sign. Since this is a loss to be minimized, it should be the negative log-likelihood, i.e., sum over -log P(y'_i | H, y'_{<i}).","section":"Eq. (1)"},{"comment":"The sentence 'using one 8 A100 GPUs' is grammatically unclear; please state the exact GPU count for the 3B and 8B runs.","section":"§4.1"},{"comment":"The streaming vs. offline comparison is based on a single implementation of the trigger. It would be useful to report the average number of segments per response and the distribution of segment lengths, since these directly affect the train/inference mismatch discussed above.","section":"§5.4 / Table 6"},{"comment":"The three components of the Echo training data (everyday dialogue, speech reasoning, knowledge QA) are listed without sizes or proportions. Please add the number of examples and hours for each component.","section":"§3.3"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the stress-test concern lands. The reader's '12.8-point gain over cascade' misreads Table 4; the measured gain is over the 'w/o Echo training' row, and that row is not defined. The pseudo-label construction in §2.4 is the main technical risk: it is a non-stationary self-distillation loop, and no experiment controls for the quality of X' or the freshness of Y'. These issues are fixable with additional experiments and a rewritten discussion, so I do not recommend rejection. Please also ask the authors to ensure the abstract and conclusion do not claim to beat the cascade baseline when Table 4 shows the opposite."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about this paper. First, the core idea is genuinely new: instead of training a speech decoder on ground-truth TTS codec tokens, the authors freeze a text-to-codec module, generate greedy text from the current speech-to-text model's hidden states, run that text through the frozen T2C to get pseudo speech tokens, and train a T2C-initialized decoder to predict those pseudo-labels from hidden states. That is a neat self-distillation trick, and I don't see it in the cited prior work. Second, the headline result is not what the abstract implies. On the same data (Table 4), EchoX averages 37.1 while a simple cascade baseline gets 43.2. EchoX loses to cascade. The 12.8-point gain that gets quoted is EchoX versus the 'w/o Echo training' ablation (24.3), not versus cascade. So the causal claim—that echo training mitigates the acoustic-semantic gap—rests entirely on one ablation.\n\nWhat the paper does well: the three-stage pipeline is clearly written, the data cleaning and synthesis pipeline is unusually detailed, unit language roughly halves speech token length with no accuracy loss, and the human preference evaluation is a reasonable extra. The comparison against interleaving, which collapses to 12.8 at this data scale, is informative. A repo is promised.\n\nSoft spots. The w/o-Echo ablation is not sufficiently specified. What replaces the echo loss? Is the decoder still initialized from T2C? Does it train on ground-truth TTS targets? Without that, the claimed 12.8-point gain could come from target-supervision differences rather than echo. Second, the pseudo-label loop is non-stationary: X' is the greedy decode of the current LoRA-updated model, and the paper never says whether targets are cached or refreshed. If refreshed, training chases a moving distribution; if stale, the decoder learns to mimic an outdated model's text errors. No control for X' quality. Third, there are no error bars or repeated runs; the differences in Table 4 could be run-to-run noise. Fourth, the representational analysis in Figure 6 is two word pairs—suggestive at best, not evidence that echo training changes the gap. Finally, the streaming train/inference mismatch (full hidden states in training, partial during inference) is unanalyzed; Table 6 suggests it works, but it deserves a comment.\n\nNet: the paper is a legitimate contribution with a promising training objective, but its central evidence is one under-specified ablation, and the actual headline—EchoX below cascade on the same data—undercuts the 'preserves reasoning' framing. The data-efficiency comparison to models trained on millions of hours is real though: EchoX-8B lands near MinMo and VITA on these QA sets.\n\nRecommendation: send it to peer review. The idea is worth a serious referee, and the fixes—specify the ablation, add variance estimates, address the target loop—are doable. I would not desk-reject, but the authors should be pushed on the ablation and the causal language.","headline":"Echo training is a genuinely new self-distillation trick for speech-to-speech LLMs, but the paper's central ablation is under-specified and EchoX actually trails a simple cascade on the same data.","tokens_in":17236,"tokens_out":4728,"would_cite":true,"duration_ms":54479,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"EchoX claims that speech-to-speech language models lose reasoning because acoustic and semantic objectives conflict, and that pseudo speech labels decoded from the model's own hidden states close the gap.","keywords":["speech-to-speech LLM","acoustic-semantic gap","echo training","pseudo speech labels","text-to-codec","unit language","streaming generation","knowledge-based QA"],"falsifier":"Take the released code and training data, and in Stage III replace the greedy-decoded text X' used to produce pseudo-labels with a random text sequence while keeping everything else constant. If the 12.8-point gain over the no-echo baseline persists, then the semantic anchoring claim is falsified because the content of the pseudo-labels is not what carries the result. Alternatively, an oracle version using codec tokens of the ground-truth response text should match or exceed the echo-trained model if the mechanism is simply better acoustic-semantic alignment.","tokens_in":16302,"feed_emoji":"🎤","tokens_out":6674,"duration_ms":72802,"temperature":0.7,"pith_summary":"This paper argues that speech-to-speech language models degrade in reasoning because their training objective overweights acoustic precision—matching exact pronunciations—at the expense of semantic correctness. To close this acoustic-semantic gap, EchoX trains a speech decoder on pseudo speech tokens generated from the model's own semantic hidden states rather than on independently recorded audio. The pseudo tokens come from a frozen text-to-codec module that converts the model's greedy-decoded text into speech tokens, and a denoising adapter keeps redundant acoustic noise out of the hidden representations. On knowledge-based question answering, the resulting model trained on about six thousand hours of data scores comparably to models trained on millions of hours, and an ablation shows echo training adds 12.8 average points over the same pipeline without it. The authors present this as an efficient way to make speech models retain the knowledge and reasoning of text LLMs.","feed_headline":"Echo training boosts speech-model QA by 12.8 points","feed_subtitle":"The 6k-hour EchoX model nearly matches million-hour speech LLMs on knowledge benchmarks.","key_machinery":"The load-bearing mechanism is echo training itself: a frozen text-to-codec (T2C) module that converts text into discrete speech tokens is used to produce pseudo-labels from the LLM's own greedy-decoded text, and an Echo decoder—initialized with the same parameters—is trained to predict those tokens from the LLM's hidden states. A denoising adapter (a feed-forward network) aligns the hidden states with the T2C embeddings via cosine similarity, reducing redundant acoustic information before the decoder sees it. The model also uses unit language, a compressed speech-token format obtained by statistically segmenting discrete speech units into word-like chunks, and a streaming trigger that decide","core_discovery":"The central claim is that the acoustic-semantic gap—the divergence between what a speech model must match acoustically (exact pronunciation) and what it must understand semantically (meaning)—is a main cause of the intelligence drop in speech-to-speech LLMs. EchoX bridges this gap by training the speech decoder on pseudo targets produced from the model's own semantic representations: the speech-to-text LLM generates text, a frozen text-to-codec module converts that text into speech tokens, and the Echo decoder learns to reconstruct those tokens from the hidden states. A denoising adapter removes redundant information before the hidden states enter the decoder, and a text objective is retaine","pith_inferences":["Echo training can be viewed as a form of self-distillation where the text-to-codec module is a fixed teacher that re-encodes the student's own semantic representation; this suggests the idea could transfer to other modality pairs, such as vision-to-speech or code-to-speech, whenever a frozen downstream encoder is available.","Because pseudo-labels are derived from the model's greedy-decoded text, the approach inherits any recognition errors from the speech-to-text stage; a natural testable extension is to generate pseudo-labels directly from hidden states rather than from the discrete text bottleneck, which might reduce error propagation.","The streaming trigger uses a single fixed threshold and window size; a learned policy could adapt to content complexity and may close the small gap between streaming and offline decoding seen at the 8B scale.","The paper's human evaluation shows echo training improves helpfulness more than naturalness, suggesting that combining echo training with a prosody-aware or style-controlled decoder could preserve semantic alignment while improving speech quality—an avenue the authors leave open."],"forward_implications":["Echo training makes speech-to-speech LLM training data-efficient: with roughly six thousand hours of data, EchoX achieves knowledge-based QA scores comparable to models trained on millions of hours.","The denoising adapter plus cosine alignment between hidden states and text embeddings provides a concrete, transferable recipe for reducing acoustic-semantic conflict in multimodal LLMs.","Unit language roughly halves speech token length relative to raw units while improving QA accuracy and ASR intelligibility of generated audio, easing long-sequence modeling.","The streaming trigger cuts synthesis latency by about five times with negligible accuracy loss, showing that partial hidden states can drive speech generation effectively.","The same-data ablation shows that echo training adds 12.8 average points over a same-architecture model without it, isolating the pseudo-label mechanism as the source of improvement within this pipeline."],"fun_headline_variants":["EchoX shrinks acoustic-semantic gap, lifts speech QA","Speech LLM knowledge boost via echo training","Echo training: 6k hours rival million-hour speech LLMs","Acoustic-semantic gap fix powers speech reasoning","EchoX: echo training bridges speech LLM intelligence gap"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the frozen text-to-codec pseudo-labels faithfully reflect the semantic content of the LLM's hidden states; if those labels are noisy, or if the hidden states seen during streaming inference differ from full-sequence training states, the echo decoder learns to replicate that noise and the claimed semantic alignment collapses.","fun_headline_variants_meta":{"raw":{"variants":["EchoX shrinks acoustic-semantic gap, lifts speech QA","Speech LLM knowledge boost via echo training","Echo training: 6k hours rival million-hour speech LLMs","Acoustic-semantic gap fix powers speech reasoning","EchoX: echo training bridges speech LLM intelligence gap"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000736,"raw_usage":{"total_tokens":3082,"prompt_tokens":659,"completion_tokens":2423,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":403,"completion_tokens_details":{"reasoning_tokens":2341}},"tokens_in":403,"tokens_out":2423,"duration_ms":17389,"temperature":1.0,"reasoning_tokens":2341,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T19:32:19.967668+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the released code and training data, and in Stage III replace the greedy-decoded text X' used to produce pseudo-labels with a random text sequence while keeping everything else constant. If the 12.8-point gain over the no-echo baseline persists, then the semantic anchoring claim is falsified because the content of the pseudo-labels is not what carries the result. Alternatively, an oracle version using codec tokens of the ground-truth response text should match or exceed the echo-trained model if the mechanism is simply better acoustic-semantic alignment.","supporting_citations":[],"review_version":1}