Pith. sign in

REVIEW 3 major objections 6 minor 75 references

SimulS2S-LLM: Unlocking Simultaneous Inference of Speech LLMs for Speech-to-Speech Translation

T0 review · 3 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read An offline-trained speech LLM can be made to stream speech-to-speech translation by feeding it boundary-aware speech prompts from CIF and decoding with a test-time wait-k policy; on CVSS-C it beats existing same-data simultaneous systems…

desk verdict First LLM-based simultaneous speech-to-speech translation system with a clean recipe and honest limitations; the headline gain over StreamSpeech is real but partly confounded by LLM text pretraining, and the CIF boundary assumption deserves a direct diagnostic. read the letter →

arxiv 2504.15509 v1 pith:ODNUGYYP submitted 2025-04-22 cs.CL cs.SDeess.AS

Keqi Deng , Wenxi Chen , Xie Chen , Philip C. Woodland This is my paper · ORCID
classification cs.CLcs.SDeess.AS
keywords simultaneousspeech-to-speechtranslationspeechlargelanguagemodelscontinuousintegrate-and-firewait-kpolicydiscretetokensstreaminginferenceCVSScorpusboundary-awareprompts
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SimulS2S-LLM sets out to show that a speech-conditioned large language model trained offline on complete utterances can still translate speech to speech while the input is arriving. The enabling idea is to feed the frozen LLM a boundary-aware speech prompt built by continuous integrate-and-fire, so the number of prompt vectors available from a chunk of audio tracks how many source-language tokens have been spoken. At test time a wait-k policy keeps LLM generation k tokens behind the growing prompt, and discrete speech tokens are decoded from a weighted sum of the LLM's hidden states and turned into audio by a vocoder. On CVSS-C Spanish-English, French-English, and German-English the paper reports a better quality-latency trade-off than existing simultaneous speech-to-speech methods trained on the same data, for example about 3 ASR-BLEU points higher at similar latency. If the claim is right, a single offline-trained speech LLM can serve both offline and streaming tasks without retraining.

What carries the argument

The carrying object is the boundary-aware speech prompt extracted by CIF (continuous integrate-and-fire). For each encoder frame $e_t$ a scalar alpha_t is accumulated from left to right; when the running sum passes 1.0 the current weight is split so that exactly 1.0 fires one prompt vector $p_i$, the weighted frame sum is emitted, and accumulation resets. Because the CIF is trained with a quantity loss matching total accumulated weight to the source text length $N$, the number of prompt vectors read from a partial audio chunk is a proxy for how many source tokens have been spoken, which is what lets the offline-trained, frozen LLM treat the partial prompt like a text prefix. This makes the test-time rule $L_{gen} = L_p - L_{prev} - K + 1$ (Algorithm 1) work: the LLM stays $K$ tokens behind the prompt, exactly as in text wait-k simultaneous translation.

What would settle it

Take the trained system and perturb the CIF firing times on the first chunks, shifting or dropping a fraction of prompt boundaries, and measure the ASR-BLEU change; if a small perturbation erases the advantage over StreamSpeech, then CIF boundary accuracy is the causal ingredient. A complementary check is to compute alignment error between CIF firing times on streaming prefixes and oracle word boundaries on CVSS-C test audio.

Watch

Extended reading notes

Core claim

The paper's central claim is that the obstacle to streaming a speech LLM is not the decoder-only architecture itself but the mismatch between the full-speech prompt seen in offline training and the partial prompt available during simultaneous inference. SimulS2S-LLM removes that mismatch with CIF: encoder frames are weighted and integrated left-to-right, and each time the accumulated weight reaches 1.0 the weighted frame sum fires as one prompt vector, so the prompt length sits at the source-text-token level. Inference is then driven by the rule $L_{gen} = L_p - L_{prev} - K + 1$ from Algorithm 1, which keeps the LLM exactly $K$ tokens behind the growing speech prompt and reproduces the behaviour of text-based wait-k. Output speech is generated from a weighted sum of multi-layer LLM hidden states by a causal speech generator using CTC, an incremental beam search, and a speech-token n-gram language model, with a pre-trained vocoder synthesising audio. The paper reports that on CVSS-C Es-En, Fr-En, and De-En this gives a better ASR-BLEU-versus-latency trade-off than StreamSpeech and than a boundary-unaware downsampling baseline, with the boundary-aware prompts contributing roughly 4 BLEU points at matched latency.

Load-bearing premise

The method rests on the assumption that CIF, when it only sees part of the audio, still counts out prompt vectors whose cumulative length tracks the semantic prefix of the source, so the frozen LLM is always handed the right prefix; if boundary counts drift on partial input, the quality-latency gains would not transfer.

Editorial extensions

If this is right

  • Changing $K$ at test time moves the system along the quality-latency curve without retraining, so one offline-trained checkpoint can serve many latency budgets.
  • Because the CIF prompt behaves like a source-text prefix, the wait-k tail beam search, incremental beam search, and speech-token n-gram shallow fusion all apply cleanly to speech output.
  • The frozen LLM is never trained for streaming, so the same model retains whatever non-streaming capabilities the base text LLM has; streaming behaviour is an inference-time property.
  • On the three language pairs tested, SimulS2S-LLM improves ASR-BLEU by roughly 3-4 points over StreamSpeech at comparable latency, and the advantage survives computation-aware latency measurement in the appendix.
  • Scaling should be additive: the paper states that using a larger text LLM or more speech-to-speech training data should further improve the generated speech quality and the trade-off.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not measure how accurate CIF boundaries are on partial chunks; a natural extension would be to compare CIF firing times against oracle source-word boundaries on streaming prefixes, since boundary error on early chunks should predict how much of the reported quality-latency gain transfers to noisier audio.
  • The recipe is not translation-specific, so the same boundary-aware prompt plus wait-k could unlock streaming for other speech-conditioned LLM tasks such as live speech continuation or speech question answering; the paper only evaluates translation and does not claim these transfers.
  • The fixed wait-k already needs larger $K$ for German-English than for Spanish/French-English, which suggests that for language pairs with heavier reordering an adaptive or learned latency policy would extend the method beyond what the paper tests.
  • The computation-aware latency in Appendix D shows the frozen LLM itself is a large part of the measured delay; as fast serving or speculative decoding improves, the same algorithm's quality-latency curve would shift left, so part of the reported trade-off is a hardware statement.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces SimulS2S-LLM, an offline-trained speech LLM system for simultaneous speech-to-speech translation (Simul-S2ST). The architecture combines a streaming acoustic encoder, a CIF module that extracts boundary-aware speech prompts, a frozen text LLM (BLOOMZ-7B or Llama3-8B), and a causal speech generator that predicts target-language discrete speech tokens. The system is trained in two stages: first a speech-to-text translation stage with cross-entropy and a CIF quantity loss, then a CTC-based speech-token stage with a weighted-sum of multi-layer LLM hidden states. At inference, a test-time wait-k policy controls the number of LLM tokens generated from each new speech chunk, and an incremental beam search with an n-gram speech-token LM is used for streaming speech-token decoding. Experiments on CVSS-C Es-En, Fr-En, and De-En report ASR-BLEU/ATD trade-offs against StreamSpeech and a boundary-unaware SimulS2S-LLM baseline, plus ablations on hidden-state layers and speech-token decoding, and BLASER 2.0 scores.

Significance. If the results hold, the paper makes a useful contribution: it is, to the best of the authors' and my knowledge, the first work to apply LLMs to Simul-S2ST while keeping the model trained offline, and it shows a clear quality-latency improvement over a boundary-unaware version of the same architecture. The strength of the paper is the internal control: the boundary-unaware SimulS2S-LLM shares the same LLM, encoder, and speech generator, so the ablation isolates the effect of CIF-based boundary-aware prompts. Evaluation uses held-out CVSS-C test sets with standard external metrics (SacreBLEU, ATD, AL, BLASER 2.0), and computation-aware latency numbers are reported in the appendix. The main risks are that the core CIF-boundary assumption is under-tested and that the headline comparison to StreamSpeech is confounded by the LLM's text pretraining.

major comments (3)
  1. [Section 3.2, Eq. (2), and Fig. 1] The symbol N is used inconsistently. In Fig. 1, N is the length of the target-language token sequence [y_1,...,y_N], while the text of Section 3.2 says the quantity loss L_qua aligns accumulated weights with the source text length N. This distinction is not cosmetic: the CIF prompt length Lp in Algorithm 1 is precisely what drives the wait-k relation Lgen = Lp - Lprev - K + 1. If the quantity loss is computed against target length rather than source length, Lp is not a source-prefix count and the core mechanism is unmoored. Please define separate symbols for source and target lengths, state explicitly which length is used in Eq. (2), and report the empirical relation between Lp and the actual number of source words read on partial inputs.
  2. [Section 3.4 and Algorithm 1] The inference procedure assumes that the CIF module, run on partial chunks of streaming speech, produces speech prompts whose length Lp equals the number of source-language semantic units read so far. The training signal, Eq. (2) plus the S2TT cross-entropy, supervises only the utterance-level total count, not per-boundary firing decisions on partial inputs, and the paper reports no diagnostic for early or late CIF firing. Since a mismatched prefix would degrade the frozen LLM's predictions, please add an oracle-prefix experiment that feeds the LLM true source prefixes of the corresponding length, or report CIF fire-time statistics against actual source word boundaries. The paper's own Limitations items 2 and 5 acknowledge scope limitations but do not address this boundary-quality assumption, which is load-bearing for the central claim.
  3. [Section 5.1, Fig. 4, and Table 1] The headline comparison against StreamSpeech is not fully controlled. StreamSpeech is trained from scratch on CVSS-C without an LLM, whereas SimulS2S-LLM uses frozen BLOOMZ-7B or Llama3-8B backbones with large text pretraining; the abstract's phrase 'same training data' does not control for pretraining. The boundary-unaware SimulS2S-LLM ablation is the appropriate control and does support the method, but the reported 3-4 point ASR-BLEU advantage over StreamSpeech could be largely attributable to the text prior rather than to the proposed simultaneous-inference mechanism. Please either add a non-LLM architecture control trained under the same recipe, or present the StreamSpeech comparison explicitly as an external benchmark and base the method-specific claim on the boundary-aware versus boundary-unaware comparison.
minor comments (6)
  1. [Section 5.1] There is a typo: 'Henceence, the extensive comparisons' should read 'Hence, the extensive comparisons'.
  2. [Appendix D] The section title 'Compution-aware Latency Results' contains a typo and should read 'Computation-aware Latency Results'.
  3. [Algorithm 1] The notation E:(n+1)∗c is ambiguous; please define E, c, n, y, Lmax, K, and Final before the pseudocode, and specify the shapes of the chunked encoder output.
  4. [Section 3.4] The statement that past key-value caches including positional information are updated in parallel before LLM generation would benefit from a concrete description of how positional encodings change when Lp grows; as written, the reader cannot verify the claimed latency behavior.
  5. [Section 4.2] The '0.5 weight' for the speech-token n-gram LM shallow fusion should be named as a hyperparameter and listed with the other hyperparameters in Appendix B.
  6. [Figure 3] The caption refers to panels (a), (b), and (c), but panel (c) is only described as an 'overall illustration'; please label the speech-prompt length and hidden-state indices in each panel so the wait-3 example is legible.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SimulS2S-LLM's reported gains are measured on held-out CVSS-C test sets with external metrics, and the CIF/wait-k mechanism is an architectural policy rather than a fitted prediction.

full rationale

The derivation chain is self-contained against external benchmarks. The central comparison, ASR-BLEU versus ATD, uses SimulEval, SacreBLEU, and the ASR-BLEU toolkit on held-out CVSS-C test data, and speech quality is additionally checked with BLASER 2.0; no result is defined in terms of a quantity fitted from that test data. The CIF quantity loss in Eq. 2 and the boundary-aware prompt construction in Section 3.2 are trained objectives and architectural choices, not re-labeled predictions. Algorithm 1's relation Lgen = Lp - Lprev - K + 1 is a test-time wait-k policy that the system follows during streaming inference, not an identity imported from the training objective. The ablation against the boundary-unaware prompt is an empirical test of the paper's design hypothesis, and the limitations sections narrow the scope while explicitly acknowledging untested conditions, which is a scope caveat rather than a circularity. Self-citations to prior work by the same group, such as Deng and Woodland (2024b) for the CIF-based design and the low-latency suitability claim, are background and design rationales and do not carry the empirical quality-latency claim. No step matching the enumerated circularity patterns can be exhibited from the paper's equations or citations, so the correct finding is no significant circularity.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The central claim rests on standard speech translation components plus three empirical assumptions: CIF alignments remain valid on partial streaming input, wait-k on prompt counts approximates prefix-to-prefix translation, and a frozen text LLM can treat projected speech prompts as text embeddings. These assumptions are partially tested by the boundary-unaware ablation and the main trade-off curves, but they are not proven and limit the claim to moderate-latency, monotone-friendly language pairs.

free parameters (7)
  • CIF integration threshold = 1.0
    Fixed threshold for weight accumulation in the CIF module (Section 3.2); not learned.
  • Quantity loss weight gamma = 0.05
    Eq. 2 uses gamma=0.05 to balance the CIF quantity loss against cross-entropy.
  • Encoder chunk size = 32 frames (~320 ms)
    Section 4.2 sets chunk size to 32 frames, controlling streaming granularity and latency.
  • Speech generator up-sampling rate U = 25
    Section 4.2: each LLM hidden state is up-sampled by 25 to produce CTC frames.
  • Incremental beam search beam size = 10
    Section 4.2: beam width for speech token prediction.
  • LLM inference beam size = 5
    Appendix B: beam size for text-token generation by the LLM.
  • N-gram LM fusion weight = 0.5
    Section 4.2: weight given to the KenLM 4-gram score in CTC decoding.
assumptions (5)
  • domain assumption CIF accumulation with a threshold of 1.0 produces one speech prompt per source text token, and the quantity loss teaches this alignment under full-utterance training.
    Section 3.2: quantity loss L_qua aligns total weights with source text length N; streaming inference assumes this alignment holds on partial chunks.
  • domain assumption Target words are produced in roughly monotonic order with respect to source speech prompts, so wait-k on CIF prompt counts is a valid prefix policy.
    Section 3.4: L_gen = L_p - L_prev - K + 1 assumes the first L_p speech prompts correspond to the translation prefix written so far; reordering-heavy language pairs are excluded.
  • domain assumption A frozen text LLM can treat FC-projected CIF speech prompts as though they were text token embeddings and produce reasonable target text.
    Section 3.3: the LLM is kept fixed and conditioned on boundary-aware speech prompts plus textual instructions.
  • domain assumption mHuBERT semantic tokens plus a unit HiFi-GAN vocoder can synthesize intelligible target speech from predicted discrete tokens.
    Sections 4.2 and 3.1: target speech is represented by mHuBERT units and synthesized by a pretrained vocoder.
  • domain assumption CTC with up-sampled LLM hidden states and an n-gram shallow fusion approximates the target speech token sequence.
    Sections 3.3 and 3.4: speech generator predicts semantic tokens under CTC, assuming conditional independence per frame.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SimulS2S-LLM: Unlocking Simultaneous Inference of Speech LLMs for Speech-to-Speech Translation." pith.science (2026). https://pith.science/paper/ODNUGYYP

@misc{pith2026250415509,
  author       = {Pith},
  title        = {Pith review of: SimulS2S-LLM: Unlocking Simultaneous Inference of Speech LLMs for Speech-to-Speech Translation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ODNUGYYP}},
  note         = {Machine review of arXiv:2504.15509}
}
read the original abstract

Simultaneous speech translation (SST) outputs translations in parallel with streaming speech input, balancing translation quality and latency. While large language models (LLMs) have been extended to handle the speech modality, streaming remains challenging as speech is prepended as a prompt for the entire generation process. To unlock LLM streaming capability, this paper proposes SimulS2S-LLM, which trains speech LLMs offline and employs a test-time policy to guide simultaneous inference. SimulS2S-LLM alleviates the mismatch between training and inference by extracting boundary-aware speech prompts that allows it to be better matched with text input data. SimulS2S-LLM achieves simultaneous speech-to-speech translation (Simul-S2ST) by predicting discrete output speech tokens and then synthesising output speech using a pre-trained vocoder. An incremental beam search is designed to expand the search space of speech token prediction without increasing latency. Experiments on the CVSS speech data show that SimulS2S-LLM offers a better translation quality-latency trade-off than existing methods that use the same training data, such as improving ASR-BLEU scores by 3 points at similar latency.

Figures

Figures reproduced from arXiv: 2504.15509 by the authors.

Figure 1
Figure 1. Illustration of SimulS2S-LLM offline training. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Illustration of the boundary-aware speech [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Illustration of the simultaneous inference of offline-trained SimulS2S-LLM (with wait-3 as an example) [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Simul-S2ST quality-latency trade-off curves [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Simul-S2TT quality-latency trade-off curves [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 8
Figure 8. Figure 8: Simul-S2ST results of different models on [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

75 extracted references · 39 canonical work pages

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. https://api.semanticscholar.org/CorpusID:257532815 GPT-4 technical report . arXiv preprint arXiv:2303.08774

    arXiv 2023

  2. [2]

    Lo \" c Barrault, Yu-An Chung, Mariano Coria Meglioli, David Dale, Ning Dong, Mark Duppenthaler, Paul-Ambroise Duquenne, Brian Ellis, Hady Elsahar, Justin Haaheim, et al. 2023. https://api.semanticscholar.org/CorpusID:266149504 Seamless: Multilingual expressive and streaming speech translation . arXiv preprint arXiv:2312.05187

    arXiv 2023

  3. [3]

    Zal \'a n Borsos, Rapha \"e l Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, et al. 2023. https://doi.org/10.1109/TASLP.2023.3288409 AudioLM : A language modeling approach to audio generation . IEEE/ACM Transactions on Audio, Speech, and Language Processing, ...

    arXiv 2023

  4. [4]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf Language models are few-shot learners . In Proc. NeurIPS, volume 33, Online

    work page 2020

  5. [5]

    Chih-Chiang Chang and Hung-yi Lee. 2022. https://doi.org/10.21437/Interspeech.2022-10627 Exploring continuous integrate-and-fire for adaptive simultaneous speech translation . In Proc. Interspeech, Incheon, Korea

    work page doi:10.21437/interspeech.2022-10627 2022

  6. [6]

    Xuankai Chang, Brian Yan, Yuya Fujita, Takashi Maekaku, and Shinji Watanabe. 2023. https://www.isca-archive.org/interspeech_2023/chang23b_interspeech.pdf Exploration of efficient end-to-end ASR using discretized input from self-supervised learning . In Proc. Interspeech , Dublin, Ireland

    work page 2023

  7. [7]

    Feilong Chen, Minglun Han, Haozhi Zhao, Qingyang Zhang, Jing Shi, Shuang Xu, and Bo Xu. 2023. https://api.semanticscholar.org/CorpusID:258558106 X-LLM : Bootstrapping advanced large language models by treating multi-modalities as foreign languages . arXiv preprint arXiv:2305.04160

    arXiv 2023

  8. [8]

    Zhehuai Chen, He Huang, Oleksii Hrinchuk, Krishna C Puvvada, Nithin Rao Koluguri, Piotr \.Z elasko, Jagadeesh Balam, and Boris Ginsburg. 2024. https://api.semanticscholar.org/CorpusID:270845464 BESTOW : Efficient and streamable speech language model with the best of two worlds in GPT and T5 . arXiv preprint arXiv:2406.19954

    arXiv 2024

Show all 75 references
  1. [9]

    Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. 2023. https://api.semanticscholar.org/CorpusID:265157993 Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models . arXiv preprint a...

    2023 arXiv

  2. [10]

    Wenqian Cui, Dianzhi Yu, Xiaoqi Jiao, Ziqiao Meng, Guangyan Zhang, Qichao Wang, Yiwen Guo, and Irwin King. 2024. https://api.semanticscholar.org/CorpusID:266598904 Recent advances in speech language models: A survey . arXiv preprint arXiv:2410.03751

    2024 arXiv

  3. [11]

    Costa-juss \`a

    David Dale and Marta R. Costa-juss \`a . 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.943 BLASER 2.0: a metric for evaluation and quality estimation of massively multilingual speech and text translation . In Proc. EMNLP (Findings) , Miami, Florida, USA

    2024 doi

  4. [12]

    Alexandre D \'e fossez, Laurent Mazar \'e , Manu Orsini, Am \'e lie Royer, Patrick P \'e rez, Herv \'e J \'e gou, Edouard Grave, and Neil Zeghidour. 2024. https://api.semanticscholar.org/CorpusID:273022979 Moshi: a speech-text foundation model for real-time dialogue . arXiv pr...

    2024 arXiv

  5. [13]

    Keqi Deng, Guangzhi Sun, and Philip C Woodland. 2025. Wav2Prompt : End-to-end speech prompt learning and task-based fine-tuning for text-based LLMs . In Proc. NAACL, Albuquerque, New Mexico, USA

    2025

  6. [14]

    Woodland

    Keqi Deng and Philip C. Woodland. 2024 a . https://doi.org/10.1109/TASLP.2024.3419421 Label-synchronous neural transducer for adaptable online E2E speech recognition . IEEE/ACM Transactions on Audio, Speech, and Language Processing, 32:3507--3516

    2024

  7. [15]

    Keqi Deng and Philip C Woodland. 2024 b . https://aclanthology.org/2024.acl-long.448.pdf Label-synchronous neural transducer for E2E simultaneous speech translation . In Proc. ACL, Bangkok, Thailand

    2024

  8. [16]

    Linhao Dong and Bo Xu. 2020. https://doi.org/10.1109/ICASSP40776.2020.9054250 CIF : Continuous integrate-and-fire for end-to-end speech recognition . In Proc. ICASSP, Barcelona, Spain

    2020

  9. [17]

    Qian Dong, Yaoming Zhu, Mingxuan Wang, and Lei Li. 2022. https://aclanthology.org/2022.acl-long.50 Learning when to translate for streaming speech . In Proc. ACL , Dublin, Ireland

    2022

  10. [18]

    Qianqian Dong, Zhiying Huang, Qi Tian, Chen Xu, Tom Ko, Yunlong Zhao, Siyuan Feng, Tang Li, Kexin Wang, Xuxin Cheng, Fengpeng Yue, Ye Bai, Xi Chen, Lu Lu, Zejun Ma, Yuping Wang, Mingxuan Wang, and Yuxuan Wang. 2024. https://api.semanticscholar.org/CorpusID:259076022 Polyvoice:...

    2024

  11. [19]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, et al. 2024. https://api.semanticscholar.org/CorpusID:271571434 The Llama 3 herd of models . arXiv preprint arXiv:2407.21783

    2024 arXiv

  12. [20]

    Qingkai Fang, Shoutao Guo, Yan Zhou, Zhengrui Ma, Shaolei Zhang, and Yang Feng. 2024. https://api.semanticscholar.org/CorpusID:272550428 Llama-omni: Seamless speech interaction with large language models . arXiv preprint arXiv:2409.06666

    2024 arXiv

  13. [21]

    Qingkai Fang, Yan Zhou, and Yang Feng. 2023. https://api.semanticscholar.org/CorpusID:263834729 Daspeech: Directed acyclic transformer for fast and high-quality speech-to-speech translation . In Proc. NeurIPS

    2023

  14. [23]

    Alex Graves. 2012. https://api.semanticscholar.org/CorpusID:17194112 Sequence transduction with recurrent neural networks . ArXiv, abs/1211.3711

    2012 arXiv

  15. [24]

    Gomez, and J \" u rgen Schmidhuber

    Alex Graves, Santiago Fern \' a ndez, Faustino J. Gomez, and J \" u rgen Schmidhuber. 2006. https://api.semanticscholar.org/CorpusID:9901844 Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks . In Proc. ICML , Pittsburgh, ...

    2006

  16. [25]

    Jiatao Gu, Graham Neubig, Kyunghyun Cho, and Victor O.K. Li. 2017. https://aclanthology.org/E17-1099 Learning to translate in real-time with neural machine translation . In Proc. EACL, Valencia, Spain

    2017

  17. [26]

    Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021. https://doi.org/10.1109/TASLP.2021.3122291 Hubert: Self-supervised speech representation learning by masked prediction of hidden units . IEEE/ACM transacti...

    2021

  18. [28]

    Hirofumi Inaguma, Shun Kiyono, Kevin Duh, Shigeki Karita, Nelson Yalta, Tomoki Hayashi, and Shinji Watanabe. 2020. https://aclanthology.org/2020.acl-demos.34 ESP net- ST : All-in-one speech translation toolkit . In Proc. ACL (demo) , Seattle, Washington, USA

    2020

  19. [29]

    Hirofumi Inaguma, Sravya Popuri, Ilia Kulikov, Peng-Jen Chen, Changhan Wang, Yu-An Chung, Yun Tang, Ann Lee, Shinji Watanabe, and Juan Pino. 2023 a . https://doi.org/10.18653/v1/2023.acl-long.872 U nit Y : Two-pass direct speech-to-speech translation with discrete units . In P...

    2023 doi

  20. [30]

    Hirofumi Inaguma, Sravya Popuri, Ilia Kulikov, Peng - Jen Chen, Changhan Wang, Yu - An Chung, Yun Tang, Ann Lee, Shinji Watanabe, and Juan Pino. 2023 b . https://aclanthology.org/2023.acl-long.872.pdf UnitY : Two-pass direct speech-to-speech translation with discrete units . I...

    2023

  21. [31]

    Ye Jia, Michelle Tadmor Ramanovich, Tal Remez, and Roi Pomerantz. 2022 a . https://api.semanticscholar.org/CorpusID:248863434 Translatotron 2: High-quality direct speech-to-speech translation with voice preservation . In Proc. ICML , Baltimore, USA

    2022

  22. [32]

    Ye Jia, Michelle Tadmor Ramanovich, Quan Wang, and Heiga Zen. 2022 b . https://api.semanticscholar.org/CorpusID:245853756 CVSS corpus and massively multilingual speech-to-speech translation . In Proc. LREC, Marseille, France

    2022

  23. [33]

    Weiss, Fadi Biadsy, Wolfgang Macherey, Melvin Johnson, Zhifeng Chen, and Yonghui Wu

    Ye Jia, Ron J. Weiss, Fadi Biadsy, Wolfgang Macherey, Melvin Johnson, Zhifeng Chen, and Yonghui Wu. 2019. https://www.isca-archive.org/interspeech_2019/jia19_interspeech.pdf Direct speech-to-speech translation with a sequence-to-sequence model . In Proc. Interspeech , Graz, Austria

    2019

  24. [34]

    Yasumasa Kano, Katsuhito Sudoh, and Satoshi Nakamura. 2022. https://api.semanticscholar.org/CorpusID:253802107 Average token delay: A latency metric for simultaneous translation . arXiv preprint arXiv:2211.13173

    2022 arXiv

  25. [35]

    Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. 2020. https://api.semanticscholar.org/CorpusID:222291664 HiFi-GAN : Generative adversarial networks for efficient and high fidelity speech synthesis . In Proc. NeurIPS, Online

    2020

  26. [36]

    Roman Koshkin, Katsuhito Sudoh, and Satoshi Nakamura. 2024 a . https://aclanthology.org/2024.emnlp-main.69 LLMs are zero-shot context-aware simultaneous translators . In Proc. EMNLP , Miami, USA

    2024

  27. [37]

    Roman Koshkin, Katsuhito Sudoh, and Satoshi Nakamura. 2024 b . https://aclanthology.org/2024.findings-emnlp.27 Transllama: Llm-based simultaneous translation system . In Proc. EMNLP (Findings) , Miami, USA

    2024

  28. [38]

    Chenyang Le, Yao Qian, Dongmei Wang, Long Zhou, Shujie Liu, Xiaofei Wang, Midia Yousefi, Yanmin Qian, Jinyu Li, Sheng Zhao, et al. 2024. https://api.semanticscholar.org/CorpusID:270068021 TransVIP : Speech to speech translation system with voice and isochrony preservation . ar...

    2024 arXiv

  29. [39]

    Ann Lee, Peng-Jen Chen, Changhan Wang, Jiatao Gu, Sravya Popuri, Xutai Ma, Adam Polyak, Yossi Adi, Qing He, Yun Tang, Juan Pino, and Wei-Ning Hsu. 2022. https://doi.org/10.18653/v1/2022.acl-long.235 Direct speech-to-speech translation with discrete units . In Proc. ACL, Dublin...

    2022 doi

  30. [40]

    Dan Liu, Mengge Du, Xiaoxi Li, Ya Li, and Enhong Chen. 2021. https://aclanthology.org/2021.emnlp-main.4 Cross attention augmented transducer networks for simultaneous translation . In Proc. EMNLP , Punta Cana, Dominican Republic

    2021

  31. [41]

    Danni Liu, Gerasimos Spanakis, and Jan Niehues. 2020. https://www.isca-archive.org/interspeech_2020/liu20s_interspeech.pdf Low-latency sequence-to-sequence speech recognition and translation by partial hypothesis selection . In Proc. Interspeech, Shanghai, China

    2020

  32. [42]

    Mingbo Ma, Liang Huang, Hao Xiong, Renjie Zheng, Kaibo Liu, Baigong Zheng, Chuanqiang Zhang, Zhongjun He, Hairong Liu, Xing Li, Hua Wu, and Haifeng Wang. 2018. https://aclanthology.org/P19-1289 STACL : Simultaneous translation with implicit anticipation and controllable latenc...

    2018

  33. [43]

    Xutai Ma, Mohammad Javad Dousti, Changhan Wang, Jiatao Gu, and Juan Pino. 2020 a . https://aclanthology.org/2020.emnlp-demos.19 SIMULEVAL : An evaluation toolkit for simultaneous translation . In Proc. EMNLP (Demos) , Online

    2020

  34. [44]

    Xutai Ma, Juan Miguel Pino, James Cross, Liezl Puzon, and Jiatao Gu. 2020 b . https://openreview.net/pdf?id=Hyg96gBKPS Monotonic multihead attention . In Proc. ICLR, Online

    2020

  35. [45]

    Xutai Ma, Juan Miguel Pino, and Philipp Koehn. 2020 c . https://api.semanticscholar.org/CorpusID:226245979 SimulMT to SimulST : Adapting simultaneous text translation to end-to-end simultaneous speech translation . In Proc. AACL/IJCNLP , Suzhou, China

    2020

  36. [46]

    Zhengrui Ma, Yang Feng, and Min Zhang. 2024 a . https://api.semanticscholar.org/CorpusID:274281450 Learning monotonic attention in transducer for streaming generation . arXiv preprint arXiv:2411.17170

    2024 arXiv

  37. [47]

    Ziyang Ma, Yakun Song, Chenpeng Du, Jian Cong, Zhuo Chen, Yuping Wang, Yuxuan Wang, and Xie Chen. 2024 b . https://api.semanticscholar.org/CorpusID:271710060 Language model can listen while speaking . arXiv preprint arXiv:2408.02622

    2024 arXiv

  38. [48]

    Ziyang Ma, Guanrou Yang, Yifan Yang, Zhifu Gao, Jiaming Wang, Zhihao Du, Fan Yu, Qian Chen, Siqi Zheng, Shiliang Zhang, et al. 2024 c . https://api.semanticscholar.org/CorpusID:267657710 An embarrassingly simple approach for llm with strong asr capacity . arXiv preprint arXiv:...

    2024 arXiv

  39. [49]

    Nakamura, K

    S. Nakamura, K. Markov, H. Nakaiwa, G. Kikui, H. Kawai, T. Jitsuhiro, J.-S. Zhang, H. Yamamoto, E. Sumita, and S. Yamamoto. 2006. https://doi.org/10.1109/TSA.2005.860774 The ATR multilingual speech-to-speech translation system . IEEE Transactions on Audio, Speech, and Language...

    2006

  40. [50]

    Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. https://aclanthology.org/N19-4009 Fairseq: a fast, extensible toolkit for sequence modeling . In Proc. NAACL-HLT (Demonstrations) , Minneapolis, Minnesota

    2019

  41. [51]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. https://openreview.net/forum?id=TG8KACxEON Training language models to follow instructions with human feedback . Proc. NeurIPS

    2022

  42. [52]

    Sara Papi, Marco Gaido, Matteo Negri, and Marco Turchi. 2022. https://aclanthology.org/2022.autosimtrans-1.2.pdf Over-generation cannot be rewarded: Length-adaptive average lagging for simultaneous speech translation . In Proc. 3rd AutoSimTrans, Online

    2022

  43. [53]

    Sara Papi, Matteo Negri, and Marco Turchi. 2023 a . https://aclanthology.org/2023.acl-long.745 Attention as a guide for simultaneous speech translation . In Proc. ACL , Toronto, Canada

    2023

  44. [54]

    Sara Papi, Marco Turchi, and Matteo Negri. 2023 b . https://www.isca-archive.org/interspeech_2023/papi23_interspeech.pdf AlignAtt : Using attention-based audio-translation alignments as a guide for simultaneous speech translation . In Proc. Interspeech, Dublin, Ireland

    2023

  45. [55]

    Sravya Popuri, Peng - Jen Chen, Changhan Wang, Juan Pino, Yossi Adi, Jiatao Gu, Wei - Ning Hsu, and Ann Lee. 2022. https://www.isca-archive.org/interspeech_2022/popuri22_interspeech.pdf Enhanced direct speech-to-speech translation using self-supervised pre-training and data au...

    2022

  46. [56]

    Matt Post. 2018. https://api.semanticscholar.org/CorpusID:262093697 A call for clarity in reporting BLEU scores . In WMT , pages 186--191. Association for Computational Linguistics

    2018

  47. [57]

    Yi Ren, Jinglin Liu, Xu Tan, Chen Zhang, Tao Qin, Zhou Zhao, and Tie-Yan Liu. 2020. https://aclanthology.org/2020.acl-main.350 SimulSpeech : End-to-end simultaneous speech to text translation . In Proc. ACL , Online

    2020

  48. [58]

    Chitwan Saharia, William Chan, Saurabh Saxena, and Mohammad Norouzi. 2020. https://api.semanticscholar.org/CorpusID:215786391 Non-autoregressive machine translation with latent alignments . In Proc. EMNLP , online

    2020

  49. [59]

    Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili \'c , Daniel Hesslow, Roman Castagn \'e , Alexandra Sasha Luccioni, Fran c ois Yvon, et al. 2022. https://api.semanticscholar.org/CorpusID:253420279 Bloom: A 176b-parameter open-access multilingual languag...

    2022 arXiv

  50. [60]

    Frank Seide, Morrie Doulaty, Yangyang Shi, Yashesh Gaur, Junteng Jia, and Chunyang Wu. 2024. https://api.semanticscholar.org/CorpusID:270521816 Speech ReaLLM -- real-time streaming speech recognition with multimodal LLMs by teaching the flow of time . arXiv preprint arXiv:2406.09569

    2024 arXiv

  51. [61]

    Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun MA, and Chao Zhang. 2024. https://openreview.net/forum?id=14rn7HpKVk SALMONN : Towards generic hearing abilities for large language models . In Proc. ICLR, Vienna, Austria

    2024

  52. [62]

    Yun Tang, Anna Sun, Hirofumi Inaguma, Xinyue Chen, Ning Dong, Xutai Ma, Paden Tomasello, and Juan Pino. 2023. https://aclanthology.org/2023.acl-long.695 Hybrid transducer and attention based encoder-decoder modeling for speech-to-text tasks . In Proc. ACL , Toronto, Canada

    2023

  53. [63]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. https://api.semanticscholar.org/CorpusID:259950998 LLaMa 2 : Open foundation and fine-tuned chat models . ar...

    2023 arXiv

  54. [64]

    Emiru Tsunoo, Hayato Futami, Yosuke Kashiwagi, Siddhant Arora, and Shinji Watanabe. 2024. https://api.semanticscholar.org/CorpusID:270702630 Decoder-only architecture for streaming end-to-end speech recognition . arXiv preprint arXiv:2406.16107

    2024 arXiv

  55. [65]

    Changhan Wang, Anne Wu, Jiatao Gu, and Juan Pino. 2021. https://doi.org/10.21437/Interspeech.2021-2027 CoVoST 2 and massively multilingual speech translation. In Proc. Interspeech, Brno, Czech Republic

    2021 doi

  56. [66]

    Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. 2023. https://api.semanticscholar.org/CorpusID:255440307 Neural codec language models are zero-shot text to speech synthesizers . arXiv preprint arXi...

    2023 arXiv

  57. [67]

    Jian Wu, Yashesh Gaur, Zhuo Chen, Long Zhou, Yimeng Zhu, Tianrui Wang, Jinyu Li, Shujie Liu, Bo Ren, Linquan Liu, and Yu Wu. 2023. https://doi.org/10.1109/ASRU57964.2023.10389705 On decoder-only architecture for speech-to-text and large language model integration . In Proc. AS...

    2023

  58. [68]

    Zhifei Xie and Changqiao Wu. 2024. https://api.semanticscholar.org/CorpusID:272146286 Mini-omni: Language models can hear, talk while thinking in streaming . arXiv preprint arXiv:2408.16725

    2024 arXiv

  59. [69]

    Jian Xue, Peidong Wang, Jinyu Li, Matt Post, and Yashesh Gaur. 2022. https://api.semanticscholar.org/CorpusID:248118691 Large-scale streaming end-to-end speech translation with neural transducers . In Proc. Interspeech, Incheon, Korea

    2022

  60. [70]

    Wenyi Yu, Changli Tang, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. 2024. https://doi.org/10.1109/ICASSP48485.2024.10445874 Connecting speech encoder and large language model for ASR . In Proc. ICASSP, Seoul, Korea

    2024

  61. [71]

    Xingshan Zeng, Liangyou Li, and Qun Liu. 2021. https://aclanthology.org/2021.findings-acl.218 R eal T ran S : End-to-end simultaneous speech translation with convolutional weighted-shrinking transformer . In Proc. ACL/IJCNLP (Findings) , Online

    2021

  62. [72]

    Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, et al. 2024. https://api.semanticscholar.org/CorpusID:267750101 AnyGPT : Unified multimodal LLM with discrete sequence modeling . arXiv preprint arXiv:2402.12226

    2024 arXiv

  63. [73]

    Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.1055 S peech GPT : Empowering large language models with intrinsic cross-modal conversational abilities . In Proc. EMNLP (Findings) , Singapore

    2023 doi

  64. [74]

    Shaolei Zhang, Qingkai Fang, Shoutao Guo, Zhengrui Ma, Min Zhang, and Yang Feng. 2024. https://doi.org/10.18653/v1/2024.acl-long.485 S tream S peech: Simultaneous speech-to-speech translation with multi-task learning . In Proc. ACL, Bangkok, Thailand

    2024 doi

  65. [75]

    Jinzheng Zhao, Niko Moritz, Egor Lakomkin, Ruiming Xie, Zhiping Xiu, Katerina Zmolikova, Zeeshan Ahmed, Yashesh Gaur, Duc Le, and Christian Fuegen. 2024. https://api.semanticscholar.org/CorpusID:273163354 Textless streaming speech-to-speech translation using semantic speech to...

    2024 arXiv

  66. [76]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...

  67. [77]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.