Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

EchoX: Towards Mitigating Acoustic-Semantic Gap via Echo Training for Speech-to-Speech LLMs

T0 review · 4 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read EchoX claims that speech-to-speech language models lose reasoning because acoustic and semantic objectives conflict, and that pseudo speech labels decoded from the model's own hidden states close the gap.

desk verdict Echo training is a genuinely new self-distillation trick for speech-to-speech LLMs, but the paper's central ablation is under-specified and EchoX actually trails a simple cascade on the same data. read the letter →

arxiv 2509.09174 v1 pith:4G3FMJ5U submitted 2025-09-11 cs.CL cs.AIcs.SD

Yuhao Zhang , Yuhao Du , Zhanchen Dai , Xiangnan Ma , Kaiqi Kou , Benyou Wang , Haizhou Li This is my paper · ORCID
classification cs.CLcs.AIcs.SD
keywords speech-to-speechLLMacoustic-semanticgapechotrainingpseudospeechlabelstext-to-codecunitlanguagestreaminggenerationknowledge-basedQA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that speech-to-speech language models degrade in reasoning because their training objective overweights acoustic precision—matching exact pronunciations—at the expense of semantic correctness. To close this acoustic-semantic gap, EchoX trains a speech decoder on pseudo speech tokens generated from the model's own semantic hidden states rather than on independently recorded audio. The pseudo tokens come from a frozen text-to-codec module that converts the model's greedy-decoded text into speech tokens, and a denoising adapter keeps redundant acoustic noise out of the hidden representations. On knowledge-based question answering, the resulting model trained on about six thousand hours of data scores comparably to models trained on millions of hours, and an ablation shows echo training adds 12.8 average points over the same pipeline without it. The authors present this as an efficient way to make speech models retain the knowledge and reasoning of text LLMs.

What carries the argument

The load-bearing mechanism is echo training itself: a frozen text-to-codec (T2C) module that converts text into discrete speech tokens is used to produce pseudo-labels from the LLM's own greedy-decoded text, and an Echo decoder—initialized with the same parameters—is trained to predict those tokens from the LLM's hidden states. A denoising adapter (a feed-forward network) aligns the hidden states with the T2C embeddings via cosine similarity, reducing redundant acoustic information before the decoder sees it. The model also uses unit language, a compressed speech-token format obtained by statistically segmenting discrete speech units into word-like chunks, and a streaming trigger that decide

What would settle it

Take the released code and training data, and in Stage III replace the greedy-decoded text X' used to produce pseudo-labels with a random text sequence while keeping everything else constant. If the 12.8-point gain over the no-echo baseline persists, then the semantic anchoring claim is falsified because the content of the pseudo-labels is not what carries the result. Alternatively, an oracle version using codec tokens of the ground-truth response text should match or exceed the echo-trained model if the mechanism is simply better acoustic-semantic alignment.

Watch

Extended reading notes

Core claim

The central claim is that the acoustic-semantic gap—the divergence between what a speech model must match acoustically (exact pronunciation) and what it must understand semantically (meaning)—is a main cause of the intelligence drop in speech-to-speech LLMs. EchoX bridges this gap by training the speech decoder on pseudo targets produced from the model's own semantic representations: the speech-to-text LLM generates text, a frozen text-to-codec module converts that text into speech tokens, and the Echo decoder learns to reconstruct those tokens from the hidden states. A denoising adapter removes redundant information before the hidden states enter the decoder, and a text objective is retaine

Load-bearing premise

The load-bearing premise is that the frozen text-to-codec pseudo-labels faithfully reflect the semantic content of the LLM's hidden states; if those labels are noisy, or if the hidden states seen during streaming inference differ from full-sequence training states, the echo decoder learns to replicate that noise and the claimed semantic alignment collapses.

Editorial extensions

If this is right

  • Echo training makes speech-to-speech LLM training data-efficient: with roughly six thousand hours of data, EchoX achieves knowledge-based QA scores comparable to models trained on millions of hours.
  • The denoising adapter plus cosine alignment between hidden states and text embeddings provides a concrete, transferable recipe for reducing acoustic-semantic conflict in multimodal LLMs.
  • Unit language roughly halves speech token length relative to raw units while improving QA accuracy and ASR intelligibility of generated audio, easing long-sequence modeling.
  • The streaming trigger cuts synthesis latency by about five times with negligible accuracy loss, showing that partial hidden states can drive speech generation effectively.
  • The same-data ablation shows that echo training adds 12.8 average points over a same-architecture model without it, isolating the pseudo-label mechanism as the source of improvement within this pipeline.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Echo training can be viewed as a form of self-distillation where the text-to-codec module is a fixed teacher that re-encodes the student's own semantic representation; this suggests the idea could transfer to other modality pairs, such as vision-to-speech or code-to-speech, whenever a frozen downstream encoder is available.
  • Because pseudo-labels are derived from the model's greedy-decoded text, the approach inherits any recognition errors from the speech-to-text stage; a natural testable extension is to generate pseudo-labels directly from hidden states rather than from the discrete text bottleneck, which might reduce error propagation.
  • The streaming trigger uses a single fixed threshold and window size; a learned policy could adapt to content complexity and may close the small gap between streaming and offline decoding seen at the 8B scale.
  • The paper's human evaluation shows echo training improves helpfulness more than naturalness, suggesting that combining echo training with a prosody-aware or style-controlled decoder could preserve semantic alignment while improving speech quality—an avenue the authors leave open.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. EchoX proposes a three-stage training pipeline for speech-to-speech LLMs. Stage I trains a speech-to-text LLM, Stage II trains a frozen text-to-codec (T2C) module, and Stage III introduces an 'Echo decoder' initialized from the T2C parameters. During Stage III, the S2T LLM's hidden states are greedy-decoded into text, passed through the frozen T2C module to produce pseudo speech-token targets Y', and the Echo decoder is trained to predict Y' from the hidden states, with a denoising adapter and a cosine-similarity loss. The model also uses 'unit language' speech tokens and a streaming generation trigger. Experiments on Llama Questions, Web Questions, and TriviaQA report speech-to-speech averages of 37.1 for EchoX-3B and 46.3 for EchoX-8B, competitive with models trained on much more data. The controlled comparison in Table 4 shows a 12.8-point gain over an under-specified 'EchoX w/o Echo training' condition, but EchoX remains 6.1 points below the Cascade condition.

Significance. If the central effect is real, the paper introduces a practical way to train speech-output LLMs without requiring parallel ground-truth audio for every response: the frozen T2C module acts as a target generator, and the denoising adapter is simple. The reported results with ~6k hours of training data, alongside a released code repository and an online human preference evaluation, make this a potentially useful contribution to the growing SLLM literature. However, the paper's headline claim is not supported by the controlled comparison as presented: the positive ablation is versus a poorly specified 'w/o Echo training' row, not versus the Cascade row. The pseudo-labeling loop also raises non-stationarity and self-distillation concerns that are not addressed. The significance therefore depends on whether the authors can clarify the ablation and provide evidence about target quality and target freshness.

major comments (4)
  1. [Table 4 / §5.1] The central comparison is mis-stated. EchoX-3B averages 37.1, which is 6.1 points below the Cascade row (43.2). The 12.8-point difference is between EchoX and 'EchoX w/o Echo training' (24.3), not 'over the cascade baseline' as claimed in the abstract and §5.1. Please specify exactly what the 'EchoX w/o Echo training' condition is: does it use the same Echo decoder and denoising adapter but train on ground-truth TTS codec targets? Is the decoder still initialized from T2C? What replaces L_Echo? The Cascade row is also unspecified: is it the same T2C/vocoder applied to the Stage-I text output? If the control condition uses ground-truth speech targets, then the result that pseudo-labels outperform ground-truth targets is surprising and needs a mechanistic explanation. As written, the load-bearing controlled claim rests on an ablation whose setup is not described.
  2. [§2.4, Eqs. (1)–(2)] The pseudo-label Y' is generated by greedy-decoding X' from the current S2T LLM and passing X' through the frozen T2C. Because the LoRA parameters are being updated during Stage III, X' and hence Y' are non-stationary. The paper does not state whether Y' is cached or recomputed each step. If cached, the Echo decoder is trained against a stale model's text; if recomputed, the training target drifts. More fundamentally, Y' is a self-generated target, so the Echo loss is a form of self-distillation: it may teach the decoder to speak the model's own text errors rather than to align with an external semantic ground truth. Please report an ablation on target freshness (e.g., cached vs. refreshed pseudo-labels) and an analysis of X' quality (e.g., WER or QA accuracy of the greedy-decoded text). A comparison against training the Echo decoder with ground-truth ⟨text, codec⟩ targets would directly
  3. [§2.4 vs. §2.6] Stage III trains the Echo decoder on the full response hidden sequence H = {h1,...,hn}, but streaming inference feeds only partial hidden subsequences to the Echo decoder, as decided by the cosine-similarity trigger and local-extremum criterion. This train/inference mismatch is not analyzed. Table 6 shows that streaming decoding is not catastrophic, but the paper does not explain why the decoder, trained on full sequences, transfers to truncated inputs. If the streaming trigger is purely heuristic, its interaction with the Echo decoder's autoregressive conditioning should be tested, for example by fine-tuning on randomly segmented hidden states or by ablating the trigger design.
  4. [§5.2 / Figure 6] The 'acoustic-semantic gap' evidence in Figure 6 is under-specified. The figure reports similarity between 'Hi'/'Hello' and 'Hi'/'High' for the S2T model, the interleaving model, and speech tokens, but no details are given about which hidden layer is used, how word-level representations are pooled, how many utterances are averaged, or whether the differences are stable across runs. As it stands, this is an illustrative correlation, not support for the causal claim that the acoustic-semantic gap is the main cause of knowledge degradation. Please provide the computation details and, ideally, error bars or a larger sample.
minor comments (5)
  1. [Tables 2–4] The header 'A vg.' should be 'Avg.' Also, Table 3 omits TriviaQA for LLaMA-Omni2 rows; please indicate whether this is 'not reported' or 'not evaluated'.
  2. [Eq. (1)] Eq. (1) defines L_Echo as a log-probability without a negative sign. Since this is a loss to be minimized, it should be the negative log-likelihood, i.e., sum over -log P(y'_i | H, y'_{<i}).
  3. [§4.1] The sentence 'using one 8 A100 GPUs' is grammatically unclear; please state the exact GPU count for the 3B and 8B runs.
  4. [§5.4 / Table 6] The streaming vs. offline comparison is based on a single implementation of the trigger. It would be useful to report the average number of segments per response and the distribution of segment lengths, since these directly affect the train/inference mismatch discussed above.
  5. [§3.3] The three components of the Echo training data (everyday dialogue, speech reasoning, knowledge QA) are listed without sizes or proportions. Please add the number of examples and hours for each component.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the Echo training target, though self-generated, is produced by an externally pre-trained frozen T2C module, and all headline claims are evaluated on external QA benchmarks.

full rationale

The paper's derivation chain is not circular in the sense defined here. Stage III constructs pseudo-labels Y′ by feeding the greedy text output X′ of the S2T LLM into a frozen, pre-trained text-to-codec (T2C) module (Section 2.4, Eq. 1). This is a form of self-distillation: the training target is a function of the model's own hidden states, but the T2C module itself is pre-trained in Stage II on ground-truth text–codec pairs from real synthesized speech (Section 3.2). The target is therefore grounded in external acoustic data, not defined solely by the final SLLM. The denoising loss (Eq. 2) aligns hidden states to the embedding of X′, again using the model's own text output, but this is a regularizer, not a claim that the model 'predicts' its own label in the sense of circular reasoning. The central performance claims are evaluated on external benchmarks (Llama Questions, Web Questions, TriviaQA) against models trained on other data, so the result is independently testable. The paper's use of self-citations (Soundwave for the S2T adapter, unit language for the speech token) is architectural borrowing, not a load-bearing uniqueness theorem or an ansatz smuggled in to force the conclusion. The ablation 'EchoX w/o Echo training' is not fully described, and the reader's headline misreads Table 4 (EchoX at 37.1 is 6.1 points below Cascade at 43.2; the 12.8-point gap is versus the under-specified w/o-Echo ablation at 24.3). This is a reporting/evidence concern, not a circularity: the claimed gain could in principle come from architecture or training details rather than from Echo training, but no equation or citation reduces the result to its inputs. The paper itself acknowledges limitations in Appendix C (naturalness lags stronger TTS models), which further confirms the claims are not tautological. Overall, the derivation is self-contained and the scores are 0 for circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 2 invented entities

The method rests on a small set of hand-chosen hyperparameters and strong assumptions about the quality of self-generated training targets. The main architectural inventions (Echo decoder, denoising adapter) are empirically validated, but their individual contributions are not fully isolated.

free parameters (4)
  • lambda (denoising loss weight) = 0.2
    Chosen by hand to balance LDenoising with other losses (Section 2.4); no sensitivity analysis is provided.
  • streaming threshold = 0.1
    Cosine similarity threshold for write operations in streaming (Section 2.6); no sensitivity analysis.
  • streaming window w = 5
    Local extremum window size for trigger operations (Section 2.6); no sensitivity analysis.
  • HuBERT hidden layer for unit extraction = 11th layer
    Selected hidden layer for extracting speech units (Section 2.5); the choice is not justified by experiments.
assumptions (4)
  • domain assumption The frozen T2C module produces accurate and sufficient speech targets for text.
    Stage III uses T2C-generated pseudo-labels Y' as ground truth; if these are poor, echo training is undermined (Section 2.4).
  • domain assumption Cosine similarity between adapter output and text embedding captures the semantic alignment needed for speech generation.
    LDenoising uses cosine similarity to align hidden states H with embeddings of X' (Equation 2); no evidence is given that this metric is adequate.
  • domain assumption The period embedding is a good trigger for semantic completeness in streaming.
    Streaming writes when cosine similarity to the period embedding is a local maximum (Section 2.6).
  • ad hoc to paper The acoustic-semantic gap is the main cause of knowledge degradation in SLLMs.
    This is the paper's motivating hypothesis (Introduction); it is supported only by correlational analysis (Figure 6), not a controlled causal experiment.
invented entities (2)
  • Echo decoder independent evidence
    purpose: Generates speech tokens from hidden states; initialized from T2C parameters and fine-tuned with the echo loss.
    Validated by ablations in Table 4; external replication could falsify its effectiveness.
  • Denoising adapter independent evidence
    purpose: Aligns hidden states with the text embedding space before feeding the Echo decoder.
    Its contribution is part of the loss design, but no dedicated ablation isolates its individual effect.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EchoX: Towards Mitigating Acoustic-Semantic Gap via Echo Training for Speech-to-Speech LLMs." pith.science (2026). https://pith.science/paper/4G3FMJ5U

@misc{pith2026250909174,
  author       = {Pith},
  title        = {Pith review of: EchoX: Towards Mitigating Acoustic-Semantic Gap via Echo Training for Speech-to-Speech LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4G3FMJ5U}},
  note         = {Machine review of arXiv:2509.09174}
}
read the original abstract

Speech-to-speech large language models (SLLMs) are attracting increasing attention. Derived from text-based large language models (LLMs), SLLMs often exhibit degradation in knowledge and reasoning capabilities. We hypothesize that this limitation arises because current training paradigms for SLLMs fail to bridge the acoustic-semantic gap in the feature representation space. To address this issue, we propose EchoX, which leverages semantic representations and dynamically generates speech training targets. This approach integrates both acoustic and semantic learning, enabling EchoX to preserve strong reasoning abilities as a speech LLM. Experimental results demonstrate that EchoX, with about six thousand hours of training data, achieves advanced performance on multiple knowledge-based question-answering benchmarks. The project is available at https://github.com/FreedomIntelligence/EchoX.

Figures

Figures reproduced from arXiv: 2509.09174 by the authors.

Figure 1
Figure 1. Comparison of training strategies across different models. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Comparison of models using differ￾ent training data, parameters, and performance metrics. The number within each node repre￾sents the score evaluated on the Web Questions dataset (Berant et al., 2013). There are two main paradigms for building SLLMs. The first is interleaved generation (Zeng et al., 2024), which forces the model to jointly consider both acoustics and se￾mantics, but requires a large amount of traini… view at source ↗
Figure 3
Figure 3. The three training stages of EchoX. Note that streaming modules are omitted here. [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Streaming inference process. Given that speech sequences are significantly longer than their text counterparts, waiting for complete text generation before producing speech tokens would substantially increase synthesis difficulty. Therefore, applying streaming generati…
Figure 5
Figure 5. Figure 5: An example of the Speech-to-Speech data construction pipeline. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Similarity between two words within different model. We compare the similarity of word representations across dif￾ferent models in [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Comparison the speech quality based on unit and unit language. We compared the results of using unit and unit language as speech tokens in [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: Human evaluation results. D Prompt Templates for Spoken-style Normalization This section documents the nine prompt templates used in our multi-step text cleaning and rewriting pipeline (see §B). Each template corresponds to one transformation stage, ensuring that the c…
Figure 9
Figure 9. Figure 9: Screenshot of the user evaluation experiment. [PITH_FULL_IMAGE:figures/full_fig_p029_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. DialectS2S: End-to-End Speech Dialogue Modeling for Low-Resource Chinese Dialects

    cs.CL 2026-08 conditional novelty 5.0 of 10

    DialectS2S adapts an end-to-end speech dialogue model to Chinese dialects by synthesizing dialect training data and adding a self-aligned speech supervision stage that uses the model's own text predictions to build TT...

  2. X$^3$-OPD: Distilling Reasoning into Large Audio-Language Models via On-Policy Alignment

    cs.LG 2026-07 conditional novelty 5.0 of 10

    X3-OPD improves audio-grounded reasoning by training the audio student on its own rollouts with token-level teacher feedback, using a three-tier paired text-audio corpus.

Reference graph

Works this paper leans on

22 extracted references · 1 linked inside Pith · cited by 2 Pith papers

  1. [1]

    Determine if the conversation contains sensitive content (e.g., illegal, violent, pornographic, discriminatory, etc.)

  2. [2]

    Determine if the conversation is meaningless

  3. [3]

    Determine if the conversation is suitable for reading aloud. * ** Conversations that are not suitable for reading aloud include, but are not limited to: ** * Content involving code, complex mathematical formulas/proofs, structured data (e.g., tables, lists, etc.); * Content that can only be answered in written form (e.g., fill-in-the- blanks, pinyin notat...

  4. [5]

    Sorry, I cannot answer this question

    **The assistant’s response is empty, meaningless, or contains phrases like " Sorry, I cannot answer this question" due to model limitations or malfunctions .** Example: ‘‘‘ User: How’s the weather today? Assistant: Sorry, I cannot answer this question. ‘‘‘

  5. [6]

    ** Example: ‘‘‘ User: Hello Assistant: Hello User: Hello Assistant: Hello ‘‘‘

    **The conversation contains a large amount of repetitive, mechanical, meaningless exchanges. ** Example: ‘‘‘ User: Hello Assistant: Hello User: Hello Assistant: Hello ‘‘‘

  6. [7]

    SensitiveContentJudgment

    **The conversation is vague, unclear in expression, and fails to provide useful information. ** Example: ‘‘‘ User: How do you use that thing? Assistant: What thing are you referring to? User: The thing, you know. ‘‘‘ 17 **Output format requirements: ** Please strictly follow the JSON format below: ‘‘‘json { "SensitiveContentJudgment": "Contains sensitive ...

  7. [8]

    Both the user’s questions and the assistant’s responses need to be modified according to the task above

  8. [9]

    Make sure that the updated conversation does not contain any emoji or kaomoji

Show all 22 references
  1. [10]

    judgement

    Only modify the content to remove emoji or kaomoji. Keep everything else unchanged. **Output format **: Do not fabricate any false experiences or emotions. Return the updated multi- turn conversation in JSON format as shown below: * "judgement": "Contains emoji or kaomoji" or ...

  2. [11]

    The assistant expresses personal experiences, emotions, preferences, etc., which are inappropriate for an AI voice dialogue model

  3. [12]

    I used to play that game a lot when I was young

    The assistant avoids answering a direct question from the user, or provides unhelpful, evasive, or off-topic responses. If you identify any such instances, modify the assistant’s response to: * Ensure it is appropriate for an AI (without fabricating emotions, personal experien...

  4. [14]

    Make sure that the updated conversation does not contain parentheses

  5. [15]

    judgement

    Only modify the content as per the above requirements. Keep everything else unchanged. 23 **Output format **: Do not fabricate any false experiences or emotions. Return the updated multi- turn conversation in JSON format as shown below: * "judgement": "Needs modification" or "...

  6. [17]

    Make sure that the updated conversation does not contain any uncommon abbreviations

  7. [18]

    judgement

    Only modify the content as per the above requirements. Keep everything else unchanged. **Output format **: Do not fabricate any false experiences or emotions. Return the updated multi- turn conversation in JSON format as shown below: * "judgement": "Needs modification" or "No ...

    2019

  8. [19]

    Both the user’s questions and the assistant’s responses need to be modified according to the tasks above

  9. [20]

    Make sure that the updated conversation does not contain readable non-word symbols

  10. [21]

    judgement

    Only modify the content as per the above requirements. Keep everything else unchanged. **Output format **: Do not fabricate any false experiences or emotions. Return the updated multi- turn conversation in JSON format as shown below: * "judgement": "Needs modification" or "No ...

    2019

  11. [22]

    Both the user’s questions and the assistant’s responses need to be modified according to the instructions above

  12. [23]

    Ensure that the rewritten conversation contains no numbers

  13. [24]

    conversations

    Only modify the Arabic numerals according to context, and do not alter any other part of the conversation. **Output format **: Do not fabricate any false experiences or emotions. Return the modified conversation in JSON format as shown below: 27 ‘‘‘json { "conversations": [ { ...

  14. [2025]

    ran- dom access memory

    Introduces the ShareChatX dataset. Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, et al. Qwen2-audio technical report.arXiv preprint arXiv:2407.10759, 2024. Alexandre D ´efossez, Laurent Mazar ´e, Manu Ors...

    2024 arXiv

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.