Pith. sign in

REVIEW 2 major objections 5 minor 46 references

Scheduled Interleaved Speech-Text Training for Speech-to-Speech Translation with LLMs

T0 review · 2 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Fine-tuning an LLM for speech-to-speech translation is more effective when word-aligned text tokens are interleaved into the speech-unit sequence and their ratio is gradually decayed to zero during training.

desk verdict Useful interleaved fine-tuning recipe, but the scheduling claim is not supported by the only comparison that tests it. read the letter →

arxiv 2506.10299 v1 pith:ARAEYOPY submitted 2025-06-12 cs.CL cs.SDeess.AS

Hayato Futami , Emiru Tsunoo , Yosuke Kashiwagi , Yuki Ito , Hassan Shahmohammadi , Siddhant Arora , Shinji Watanabe This is my paper · ORCID
classification cs.CLcs.SDeess.AS
keywords speech-to-speechtranslationlargelanguagemodelmodalityadaptationinterleavedspeech-texttrainingdiscretespeechunitsCTCalignmentcurriculumlearningmultilingual
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a pre-trained text LLM can be adapted to speech-to-speech translation more reliably if, during fine-tuning, its training sequences mix discrete speech units with word-aligned text tokens, with the text ratio decaying from 0.9 to 0 as training proceeds. The authors argue that this scheduled interleaving eases the modality gap between text and speech, and they report consistent ASR-BLEU gains over speech-only fine-tuning across all seven CVSS language pairs tested. The gains are largest for the lowest-resource pairs (Portuguese-English and Russian-English), where translation quality roughly doubled. If the claim holds, the method offers a simple, alignment-based recipe for squeezing better S2ST performance out of limited supervised data without changing the architecture.

What carries the argument

The central mechanism is Algorithm 1, which constructs interleaved speech-text units by randomly selecting word spans (span lengths drawn from a Poisson distribution) and replacing the corresponding speech-unit spans with their BPE text tokens. The replacement is governed by a text ratio p, which the paper schedules to decay from 0.9 by 0.1 every 300 training steps, so the model starts with mostly text-like input and gradually shifts to pure speech units. Word alignments come from the CTC output of an ASR-fine-tuned w2v-BERT encoder, so no external aligner is needed. This machinery is doing the work of progressively bridging the length and representation gaps between speech and text modalities during fine-tuning.

What would settle it

Take a low-resource language pair and rerun scheduled interleaving with alignments deliberately corrupted (e.g., random shifts of span boundaries or equal-interval spans). If the ASR-BLEU gain over the speech-only baseline does not disappear or shrink substantially compared to the aligned version, then alignment accuracy is not the load-bearing condition the paper claims.

Watch

Extended reading notes

Core claim

The central claim, stated in the abstract and conclusion, is that scheduled interleaved speech-text training improves speech-to-speech translation performance of LLMs, attributable to better modality adaptation. Concretely, fine-tuning LLaMA3.2-1B with interleaved speech-text units whose text ratio decays from 0.9 to 0 outperforms fine-tuning on speech units alone in ASR-BLEU on all seven CVSS pairs, e.g. Pt-en 19.5 vs 10.3 and Ru-en 14.1 vs 6.0, while UTMOS speech quality stays roughly constant. The paper further claims the scheduling itself matters: interleaving at a constant text ratio of 0.3 is worse, and interleaving on both the input and output sides, guided by word-level CTC alignments, is necessary.

Load-bearing premise

The method assumes the CTC word alignments correctly identify, for every training utterance in every source language, which spans of speech units correspond to which words.

Editorial extensions

If this is right

  • Scheduled ILT improves ASR-BLEU over the speech-only baseline on all seven tested CVSS language pairs, with the largest gains on low-resource pairs.
  • Interleaving at a constant text ratio does not reproduce the gain; the decay schedule is a necessary ingredient.
  • Both source-side and target-side interleaving are needed; interleaving only one side loses most of the benefit.
  • Accurate word-level alignment is required; replacing speech spans at equal intervals instead of at CTC alignments drops Pt-en from 19.5 to 11.5 ASR-BLEU.
  • Speech quality, measured by UTMOS, stays high (around 4.2) across training methods, so the translation gain is not bought with degraded audio.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The method's benefit should transfer to other decoder-only speech tasks, such as spoken dialog or speech continuation, wherever a text-pretrained model must be adapted to discrete speech units with limited data.
  • The schedule hyperparameters (starting ratio 0.9, decay step 300, Poisson span length) are untested across tasks; the optimal decay may depend on language pair length ratios and data size, so the method may need per-task tuning.
  • Because the gain concentrates in low-resource pairs, a testable extension is to use the scheduled interleaving as the fine-tuning objective for zero-shot or few-shot S2ST with more languages than seen in supervised data.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes scheduled interleaved speech-text training for speech-to-speech translation (S2ST) with LLMs. The method interleaves word-aligned text tokens into speech-unit sequences at both the source and target sides, and gradually decays the text ratio from 0.9 to 0 during fine-tuning. Experiments on CVSS with LLaMA3.2-1B show consistent ASR-BLEU improvements over a speech-only baseline across seven languages, with larger gains in lower-resource languages. Ablations indicate that both-side interleaving, real CTC word alignments, and interleaving rather than masking are all important. The paper also analyzes the evolution of length and representation gaps during training.

Significance. If the central claim holds, the method provides a straightforward and effective way to adapt text-only LLMs to S2ST, which is particularly valuable for low-resource language pairs. The ablation design is thoughtful: it separates the contributions of input/output interleaving, alignment quality, and the difference between interleaving and masking. The analysis in Figure 3 offers a plausible mechanistic explanation through progressive modality adaptation. However, the specific claim that the decay schedule itself is beneficial is not yet empirically isolated, because the only non-scheduled control is confounded with average text ratio. The paper is otherwise well positioned for the speech/LLM community, and the identified issue is fixable with additional controlled experiments.

major comments (2)
  1. [Section 3.2 and Table 2] The comparison that is supposed to isolate the effect of scheduling compares 'Scheduled ILT' (p decaying from 0.9 by 0.1 per 300 steps) with 'ILT w/o scheduling' at a constant p=0.3. These two conditions differ in two ways: the presence/absence of decay and the average fraction of text tokens observed during training. The scheduled run's time-averaged text ratio is never reported, and footnote 5 states that the actual ratio exceeds p because of Algorithm 1's stopping rule, with the Poisson span-length parameter lambda left unreported. This means the constant-p=0.3 control cannot establish that gradual decay, rather than the total quantity of text exposure, drives the gains. The confound is compounded by the fact that constant p=0.3 is worse than the baseline in Fr-en, De-en, Es-en, and Ca-en (25.8 vs 28.8, 22.1 vs 27.3, 21.9 vs 33.5, 17.8 vs 23.8), so it is not a neutral reference. To support the central claim about scheduling, the authors should add a constant-p condition matched to the scheduled run's time-averaged actual text ratio, and ideally also report schedule ablations (different initial p, different decay steps, or an increasing-p schedule) to show that the direction and rate of decay matter.
  2. [Tables 2 and 3] No multiple seeds, significance tests, or confidence intervals are reported for any condition. The paper's central empirical claims are that scheduled ILT consistently improves over baseline and that scheduling is effective. The cross-language trend is consistent, but all results are single runs; observed differences of 2-8 ASR-BLEU points could be affected by training variability, especially for the smaller low-resource subsets. At least 2-3 seeds with means and standard deviations, or pairwise significance tests, should be provided for the key comparisons in Tables 2 and 3.
minor comments (5)
  1. [Section 4 / Algorithm 1] The Poisson parameter lambda used to sample span lengths in Algorithm 1 is never reported, even though it directly affects the realized text ratio and therefore the quantities compared in Table 2. Please report lambda and the empirical average text ratio for each condition.
  2. [Section 4] The total number of training steps per language is not reported, so the meaning of 'every 300 steps' for the decay schedule is unclear. Please report the number of steps or epochs for each language subset.
  3. [Section 4 / Table 3] The ablation 'Interleave without alignment' shows that using equal-interval spans degrades performance (11.5 vs 19.5 ASR-BLEU on Pt-en), but the quality of the CTC alignments for the seven languages is not assessed. Reporting word/character alignment error rates, at least for the low-resource languages, would make the reliance on alignment more transparent.
  4. [Figure 3] The representation-similarity analysis in Figure 3 is only shown for Pt-en and is described qualitatively. Reporting the numerical similarity values at the final training step, or a summary statistic, would strengthen the claim that scheduled ILT leads to better final speech-text alignment.
  5. [Abstract and Conclusion] The phrase 'especially for languages with limited training data' is supported by a visual trend in Table 2, but no correlation coefficient or other statistical evidence is provided. A simple linear regression of gains against training hours, or a scatter plot, would substantiate this claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the S2ST improvement claims are empirical, evaluated with external metrics, and none of the load-bearing steps reduces to the paper's own inputs.

full rationale

The paper is an empirical fine-tuning study. Its central claim is that scheduled interleaved speech-text training improves speech-to-speech translation, measured by Whisper ASR-BLEU and UTMOS on held-out CVSS test sets. These metrics come from external models and public corpus annotations, not from quantities defined by the proposed method. There is no derivation chain in which a prediction is equivalent to an input by construction. Eq. (3) is simply the training objective, and Algorithm 1 defines the interleaving preprocessing; neither assumes the conclusion that scheduling helps. The schedule hyperparameters (initial p=0.9, decay 0.1 per 300 steps) are hand-chosen rather than fitted to the evaluation metric, so no fitted parameter is renamed as a prediction. The paper cites AudioPaLM and SpiRitLM as prior work for context and inspiration, but those citations are not self-citations load-bearing for the current claim, and no uniqueness theorem or ansatz is imported from the authors' own prior work. The skeptical concern that the constant-p=0.3 control may not isolate gradual decay from average text exposure, especially given footnote 5's admission that the actual text ratio exceeds p, is a genuine experimental confound and a correctness risk, but it is not circularity: the comparison still tests a concrete training recipe against external metrics and does not reduce by definition to the method's inputs. No self-definitional, fitted-input, self-citation-chain, or renaming pattern is present. Therefore the honest finding is no significant circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The ledger is light because the paper is an empirical training-method study: no new entities are introduced, and the central claim is judged against external metrics (Whisper ASR-BLEU, UTMOS). What the claim rests on are the borrowed speech-unit pipeline (w2v-BERT, k-means, unit HiFi-GAN) treated as reliable, the accuracy of CTC word alignments, which the paper's own ablation shows is load-bearing, and hand-chosen training hyperparameters, notably the schedule and the constant p=0.3 baseline, which are never ablated. The Poisson lambda is unspecified.

free parameters (3)
  • Initial text ratio and decay schedule = p=0.9, decay 0.1 per 300 steps down to 0
    Hand-chosen schedule for the main method; no ablation or search over schedules is reported (Section 4, Table 2).
  • Constant interleaving ratio for ILT w/o scheduling = p=0.3
    Single hand-chosen comparison value; confounds the scheduling comparison because scheduled ILT spends most training at higher text ratios (Section 4, Table 2).
  • Poisson span length parameter (lambda) = unreported
    Algorithm 1 Line 4 samples span length from Poisson(lambda) but the paper never gives lambda, an unspecified hyperparameter that affects interleaving granularity.
assumptions (4)
  • domain assumption CTC-based word alignments from the ASR-fine-tuned w2v-BERT are accurate for all 21 source languages plus English.
    Section 3.2 uses CTC segmentation to define word spans for Algorithm 1; Table 3 shows equal-interval alignments degrade Pt-en ASR-BLEU from 19.5 to 11.5, confirming the method depends on alignment quality.
  • domain assumption Semantic units from the 20th w2v-BERT layer clustered by k-means encode the semantic content needed for translation.
    Section 4; inherited from prior speech-unit literature (GSLM, AudioPaLM), not verified in this paper.
  • domain assumption Fine-tuned LLMs can learn speech-unit input and output when vocabulary is extended, and CoT factorization does not break end-to-end optimization.
    Section 3.1, Eqs. (1)-(3); relies on AudioPaLM [7], and Table 1 checks CoT and ASR fine-tuning only on the Fr-en pair.
  • ad hoc to paper The decay schedule (start 0.9, step 0.1 per 300 steps) is a suitable curriculum for this model and data scale.
    Section 4; chosen by hand, no ablations over schedule shape, step size, or starting ratio.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Scheduled Interleaved Speech-Text Training for Speech-to-Speech Translation with LLMs." pith.science (2026). https://pith.science/paper/ARAEYOPY

@misc{pith2026250610299,
  author       = {Pith},
  title        = {Pith review of: Scheduled Interleaved Speech-Text Training for Speech-to-Speech Translation with LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ARAEYOPY}},
  note         = {Machine review of arXiv:2506.10299}
}
read the original abstract

Speech-to-speech translation (S2ST) has been advanced with large language models (LLMs), which are fine-tuned on discrete speech units. In such approaches, modality adaptation from text to speech has been an issue. LLMs are trained on text-only data, which presents challenges to adapt them to speech modality with limited speech-to-speech data. To address the training difficulty, we propose scheduled interleaved speech--text training in this study. We use interleaved speech--text units instead of speech units during training, where aligned text tokens are interleaved at the word level. We gradually decrease the ratio of text as training progresses, to facilitate progressive modality adaptation from text to speech. We conduct experimental evaluations by fine-tuning LLaMA3.2-1B for S2ST on the CVSS dataset. We show that the proposed method consistently improves the translation performances, especially for languages with limited training data.

Figures

Figures reproduced from arXiv: 2506.10299 by the authors.

Figure 1
Figure 1. Interleaved speech–text units used as input and out￾put of LLM. Text ratio p is gradually decreased as training pro￾gresses, which we call scheduled interleaved training. Due to strong language understanding capabilities of the LLM, AudioPaLM outperforms existing speech-to-text and speech-to￾speech translation systems in a wide range of languages. In this study, we build an S2ST system following AudioPaLM. In fine-t… view at source ↗
Figure 2
Figure 2. Our LLM-based speech-to-speech translation system. In experimental evaluations, we built an S2ST system by fine-tuning LLaMA3.2-1B LLM [26] on the CVSS corpus [27]. We used discrete speech units from a w2v-BERT encoder, which was fine-tuned for ASR with the CTC objective. For waveform generation, we used a unit-based HiFi-GAN vocoder [28, 29]. The overall system is shown in [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. shows how proposed scheduled ILT affects the training in terms of length and learned representations between speech and text. We used baseline and scheduled ILT models for Pt-En. The left figure shows the ratio of the speech-unit length against the text-unit length i.e. |S|/|T|. Instead of S, sched￾uled ILT adopts interleaved units Ip. We measured it for both the source and target sides. We found that the lengths si… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

46 extracted references · 43 canonical work pages

  1. [1]

    Introduction Speech-to-speech translation (S2ST) is a promising technology converts speech from one language to another, which can be useful for communicating with people who do not share the same language. Traditionally, S2ST has been solved by cas- caded approaches, which consist of automatic speech recog- nition (ASR), machine translation (MT), and tex...

    arXiv 2025

  2. [2]

    Speech-to-speech translation (S2ST) End-to-end speech-to-speech translation (S2ST) systems have been actively studied, which is jointly optimized as a speech-to- speech task [2–9]

    Related work 2.1. Speech-to-speech translation (S2ST) End-to-end speech-to-speech translation (S2ST) systems have been actively studied, which is jointly optimized as a speech-to- speech task [2–9]. Early studies have solved the task by directly predicting spectrograms in a target language with sequence-to- sequence models [2,3]. Recently, speech synthesi...

    work page

  3. [3]

    Speech-to-speech translation system As shown in Figure 2, we adopt a speech-to-speech translation (S2ST) system fine-tuned from an LLM, as in AudioPaLM [7]

    Method 3.1. Speech-to-speech translation system As shown in Figure 2, we adopt a speech-to-speech translation (S2ST) system fine-tuned from an LLM, as in AudioPaLM [7]. LetS src denote semantic units in a source language, andS tgt denote those in a target language. End-to-end S2ST is trained to predictS tgt fromS src, by minimizing the following objective...

    work page

  4. [4]

    We use the interleaved speech–text unitsI src p andI tgt p in- stead of speech semantic units during LLM fine-tuning, where pdenotes the text ratio. The Eq.(2) is rewritten withI src p and I tgt p as: L=−logp(I tgt p |T tgt, Tsrc, Isrc p ;θ) p(T tgt|T src, Isrc p ;θ)p(T src|I src p ;θ).(3) Algorithm 1 explains the detail of the interleaving algo- rithm. T...

    work page

  5. [5]

    The CVSS corpus is a widely used corpus for multilingual S2ST, built by speech synthe- sis from the CoV oST2 [36] speech-to-text translation corpus

    Experimental evaluations We conducted speech-to-speech translation (S2ST) experiments based on the CVSS [27] corpus. The CVSS corpus is a widely used corpus for multilingual S2ST, built by speech synthe- sis from the CoV oST2 [36] speech-to-text translation corpus. CVSS consists of two versions: CVSS-C with a single high- quality canonical voice and CVSS-...

    work page

  6. [6]

    We use interleaved speech–text units as the input and output of LLM, instead of the speech units, during fine-tuning LLM

    Conclusions In this study, we have proposed a new training method named scheduled interleaved speech–text training for LLM- based speech-to-speech translation (S2ST). We use interleaved speech–text units as the input and output of LLM, instead of the speech units, during fine-tuning LLM. We gradually decrease the ratio of text in interleaving, for better ...

    work page

  7. [7]

    The ATR multilingual speech-to-speech translation system,

    S. Nakamura, K. Markov, H. Nakaiwa, G. Kikui, H. Kawai, T. Jitsuhiro, J.-S. Zhang, H. Yamamoto, E. Sumita, and S. Ya- mamoto, “The ATR multilingual speech-to-speech translation system,”TASLP, 2006

    work page 2006

  8. [8]

    Direct speech-to-speech translation with a sequence- to-sequence model,

    Y . Jia, R. J. Weiss, F. Biadsy, W. Macherey, M. Johnson, Z. Chen, and Y . Wu, “Direct speech-to-speech translation with a sequence- to-sequence model,” inInterspeech, 2019

    work page 2019

Show all 46 references
  1. [9]

    Trans- latotron 2: High-quality direct speech-to-speech translation with voice preservation,

    Y . Jia, M. T. Ramanovich, T. Remez, and R. Pomerantz, “Trans- latotron 2: High-quality direct speech-to-speech translation with voice preservation,” inInternational Conference on Machine Learning, 2021

    2021

  2. [10]

    Direct speech- to-speech translation with discrete units,

    A. Lee, P.-J. Chen, C. Wang, J. Gu, S. Popuri, X. Ma, A. Polyak, Y . Adi, Q. He, Y . Tang, J. Pino, and W.-N. Hsu, “Direct speech- to-speech translation with discrete units,” inACL, 2022

    2022

  3. [11]

    UnitY: Two- pass direct speech-to-speech translation with discrete units,

    H. Inaguma, S. Popuri, I. Kulikov, P.-J. Chen, C. Wang, Y .-A. Chung, Y . Tang, A. Lee, S. Watanabe, and J. Pino, “UnitY: Two- pass direct speech-to-speech translation with discrete units,” in ACL, 2023

    2023

  4. [12]

    SeamlessM4T: Massively mul- tilingual&multimodal machine translation,

    SeamlessCommunicationet al., “SeamlessM4T: Massively mul- tilingual&multimodal machine translation,” 2023

    2023

  5. [13]

    AudioPaLM: A large language model that can speak and listen,

    P. K. Rubensteinet al., “AudioPaLM: A large language model that can speak and listen,”ArXiv, 2023

    2023

  6. [14]

    PolyV oice: Language models for speech to speech translation,

    Q. qian Dong, Z. Huang, Q. Tian, C. Xu, T. Ko, yunlong zhao, S. Feng, T. Li, K. Wang, X. Cheng, F. Yue, Y . Bai, X. Chen, L. Lu, Z. MA, Y . Wang, M. Wang, and Y . Wang, “PolyV oice: Language models for speech to speech translation,” inICLR, 2024

    2024

  7. [15]

    MSLM-S2ST: A multitask speech language model for textless speech-to-speech translation with speaker style preserva- tion,

    Y . Peng, I. Kulikov, Y . Yang, S. Popuri, H. Lu, C. Wang, and H. Gong, “MSLM-S2ST: A multitask speech language model for textless speech-to-speech translation with speaker style preserva- tion,”ArXiv, 2024

    2024

  8. [16]

    HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,

    W.-N. Hsu, B. Bolte, Y .-H. H. Tsai, K. Lakhotia, R. Salakhutdi- nov, and A. rahman Mohamed, “HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,” TASLP, 2021

    2021

  9. [17]

    w2v-BERT: Combining contrastive learning and masked language modeling for self-supervised speech pre- training,

    Y .-A. Chung, Y . Zhang, W. Han, C.-C. Chiu, J. Qin, R. Pang, and Y . Wu, “w2v-BERT: Combining contrastive learning and masked language modeling for self-supervised speech pre- training,”ASRU, 2021

    2021

  10. [18]

    SoundStream: An end-to-end neural audio codec,

    N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi, “SoundStream: An end-to-end neural audio codec,”TASLP, 2021

    2021

  11. [19]

    High fidelity neural audio compression,

    A. D ´efossez, J. Copet, G. Synnaeve, and Y . Adi, “High fidelity neural audio compression,”arXiv, 2022

    2022

  12. [20]

    Language models are few-shot learners,

    T. B. Brownet al., “Language models are few-shot learners,” ArXiv, 2020

    2020

  13. [21]

    Prompting large language models with speech recognition abilities,

    Y . Fathullah, C. Wu, E. Lakomkin, J. Jia, Y . Shangguan, K. Li, J. Guo, W. Xiong, J. Mahadeokar, O. Kalinli, C. Fuegen, and M. L. Seltzer, “Prompting large language models with speech recognition abilities,”ICASSP, 2023

    2023

  14. [22]

    SLM: Bridge the thin gap between speech and text foun- dation models,

    M. Wang, W. Han, I. Shafran, Z. Wu, C.-C. Chiu, Y . Cao, Y . Wang, N. Chen, Y . Zhang, H. Soltau, P. K. Rubenstein, L. Zilka, D. Yu, Z. Meng, G. Pundak, N. Siddhartha, J. Schalkwyk, and Y . Wu, “SLM: Bridge the thin gap between speech and text foun- dation models,”ASRU, 2023

    2023

  15. [23]

    On decoder-only architecture for speech-to-text and large language model integration,

    J. Wu, Y . Gaur, Z. Chen, L. Zhou, Y . Zhu, T. Wang, J. Li, S. Liu, B. Ren, L. Liu, and Y . Wu, “On decoder-only architecture for speech-to-text and large language model integration,” inASRU, 2023

    2023

  16. [24]

    V oxtLM: Unified decoder-only models for consolidating speech recognition, synthesis and speech, text continuation tasks,

    S. Maiti, Y . Peng, S. Choi, J. weon Jung, X. Chang, and S. Watan- abe, “V oxtLM: Unified decoder-only models for consolidating speech recognition, synthesis and speech, text continuation tasks,” ICASSP, 2023

    2023

  17. [25]

    Qwen-Audio: Advancing universal audio understanding via unified large-scale audio-language models,

    Y . Chu, J. Xu, X. Zhou, Q. Yang, S. Zhang, Z. Yan, C. Zhou, and J. Zhou, “Qwen-Audio: Advancing universal audio understanding via unified large-scale audio-language models,”ArXiv, 2023

    2023

  18. [26]

    SpeechGPT: Empowering large language models with intrinsic cross-modal conversational abilities,

    D. Zhang, S. Li, X. Zhang, J. Zhan, P. Wang, Y . Zhou, and X. Qiu, “SpeechGPT: Empowering large language models with intrinsic cross-modal conversational abilities,” inEMNLP, 2023

    2023

  19. [27]

    SALMONN: Towards generic hearing abilities for large language models,

    C. Tang, W. Yu, G. Sun, X. Chen, T. Tan, W. Li, L. Lu, Z. MA, and C. Zhang, “SALMONN: Towards generic hearing abilities for large language models,” inICLR, 2024

    2024

  20. [28]

    PaLM 2 technical report,

    R. Anilet al., “PaLM 2 technical report,”ArXiv, 2023

    2023

  21. [29]

    Bridging the modality gap for speech-to-text translation,

    Y . Liu, J. Zhu, J. Zhang, and C. Zong, “Bridging the modality gap for speech-to-text translation,”ArXiv, 2020

    2020

  22. [30]

    Push- ing the limits of zero-shot end-to-end speech translation,

    I. Tsiamas, G. G ´allego, J. Fonollosa, and M. Costa-juss `a, “Push- ing the limits of zero-shot end-to-end speech translation,” inACL, 2024

    2024

  23. [31]

    SpiRit-LM: In- terleaved spoken and written language model,

    T. Nguyen, B. Muller, B. Yu, M. R. Costa-juss `a, M. Elbayad, S. Popuri, P.-A. Duquenne, R. Algayres, R. Mavlyutov, I. Gat, G. Synnaeve, J. Pino, B. Sagot, and E. Dupoux, “SpiRit-LM: In- terleaved spoken and written language model,”TACL, 2024

    2024

  24. [32]

    The llama 3 herd of models,

    A. Dubeyet al., “The llama 3 herd of models,”ArXiv, 2024

    2024

  25. [33]

    CVSS corpus and massively multilingual speech-to-speech translation,

    Y . Jia, M. Tadmor Ramanovich, Q. Wang, and H. Zen, “CVSS corpus and massively multilingual speech-to-speech translation,” inLREC, 2022

    2022

  26. [34]

    Speech resynthesis from discrete disentangled self-supervised representations,

    A. Polyak, Y . Adi, J. Copet, E. Kharitonov, K. Lakhotia, W.-N. Hsu, A. rahman Mohamed, and E. Dupoux, “Speech resynthesis from discrete disentangled self-supervised representations,” inIn- terspeech, 2021

    2021

  27. [35]

    HiFi-GAN: Generative adversar- ial networks for efficient and high fidelity speech synthesis,

    J. Kong, J. Kim, and J. Bae, “HiFi-GAN: Generative adversar- ial networks for efficient and high fidelity speech synthesis,” in NeurIPS, 2020

    2020

  28. [36]

    On generative spoken language modeling from raw audio,

    K. Lakhotia, E. Kharitonov, W.-N. Hsu, Y . Adi, A. Polyak, B. Bolte, T.-A. Nguyen, J. Copet, A. Baevski, A. Mohamed, and E. Dupoux, “On generative spoken language modeling from raw audio,”TACL, 2021

    2021

  29. [37]

    AudioLM: A language modeling approach to audio generation,

    Z. Borsos, R. Marinier, D. Vincent, E. Kharitonov, O. Pietquin, M. Sharifi, D. Roblek, O. Teboul, D. Grangier, M. Tagliasacchi, and N. Zeghidour, “AudioLM: A language modeling approach to audio generation,”TASLP, 2023

    2023

  30. [38]

    Neural codec language models are zero-shot text to speech synthesizers,

    C. Wang, S. Chen, Y . Wu, Z.-H. Zhang, L. Zhou, S. Liu, Z. Chen, Y . Liu, H. Wang, J. Li, L. He, S. Zhao, and F. Wei, “Neural codec language models are zero-shot text to speech synthesizers,” TASLP, 2023

    2023

  31. [39]

    Speak for- eign languages with your own voice: Cross-lingual neural codec language modeling,

    Z.-H. Zhang, L. Zhou, C. Wang, S. Chen, Y . Wu, S. Liu, Z. Chen, Y . Liu, H. Wang, J. Li, L. He, S. Zhao, and F. Wei, “Speak for- eign languages with your own voice: Cross-lingual neural codec language modeling,”ArXiv, 2023

    2023

  32. [40]

    Scaling speech-text pre-training with synthetic interleaved data,

    A. Zeng, Z. Du, M. Liu, L. Zhang, S. Jiang, Y . Dong, and J. Tang, “Scaling speech-text pre-training with synthetic interleaved data,” ArXiv, 2024

    2024

  33. [41]

    CTC-segmentation of large corpora for german end-to-end speech recognition,

    L. Kurzinger, D. Winkelbauer, L. Li, T. Watzel, and G. Rigoll, “CTC-segmentation of large corpora for german end-to-end speech recognition,” inSPECOM, 2020

    2020

  34. [42]

    CoV oST 2 and massively multilingual speech translation,

    C. Wang, A. Wu, J. Gu, and J. Pino, “CoV oST 2 and massively multilingual speech translation,” inInterspeech, 2021

    2021

  35. [43]

    ESPnet: End-to-end speech processing toolkit,

    S. Watanabe, T. Hori, S. Karita, T. Hayashi, J. Nishitoba, Y . Unno, N. Enrique Yalta Soplin, J. Heymann, M. Wiesner, N. Chen, A. Renduchintala, and T. Ochiai, “ESPnet: End-to-end speech processing toolkit,” inInterspeech, 2018

    2018

  36. [44]

    Robust speech recognition via large-scale weak su- pervision,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak su- pervision,” inICML, 2022

    2022

  37. [45]

    UTMOS: Utokyo-sarulab system for voicemos challenge 2022,

    T. Saeki, D. Xin, W. Nakata, T. Koriyama, S. Takamichi, and H. Saruwatari, “UTMOS: Utokyo-sarulab system for voicemos challenge 2022,”Interspeech, 2022

    2022

  38. [46]

    Investigating decoder-only large language models for speech-to-text translation,

    C.-W. Huang, H. Lu, H. Gong, H. Inaguma, I. Kulikov, R. Mav- lyutov, and S. Popuri, “Investigating decoder-only large language models for speech-to-text translation,”Interspeech, 2024

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.