Pith. sign in

REVIEW 4 major objections 4 minor 43 references

SparQLe: Speech Queries to Text Translation Through LLMs

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SparQLe claims a Q-Former adapter can translate English speech into French and, without training on German, into German, beating IWSLT 2022 baselines on semantic similarity.

desk verdict A clean frozen-SSL-to-LLM architecture with a zero-shot result, but the headline BERTScore win rests on an evaluation protocol that the paper itself undermines. read the letter →

arxiv 2502.09284 v3 pith:7I6NBOVB submitted 2025-02-13 cs.CL cs.AI

Amirbek Djanibekov , Hanan Aldarmaki This is my paper · ORCID
classification cs.CLcs.AI
keywords speech-to-texttranslationself-supervisedspeechrepresentationsQ-Formeradapterinstruction-tunedLLMzero-shotcross-lingualtransferBERTScorespeech-LLMintegrationparameter-efficientadaptation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SparQLe is a trainable bridge that turns English audio into text an instruction-tuned LLM can read: a Q-Former adapter maps features from a frozen self-supervised speech encoder into the frozen LLM's embedding space, and only the adapter's parameters are updated. The paper reports that after pre-training on English audio and fine-tuning mainly on transcription plus English-French translation, the system scores 85.56 on BERTScore, a reference-based semantic similarity metric, on English-French speech translation, above the IWSLT 2022 strong baseline's 81.75, and 83.26 on English-German, a language never seen in fine-tuning, versus 77.44 for the strong baseline. The point of the claim is that semantic content of speech survives into an off-the-shelf text LLM without a task-specific ASR encoder, and that translation ability for new languages comes from the frozen LLM. If true, speech translation could be extended to new target languages by changing the prompt's language name, without retraining.

What carries the argument

The load-bearing component is the Q-Former modality adapter: a 12-layer transformer with 100 randomly initialized learnable query tokens, borrowed from vision-language model practice. The query tokens do not point to specific speech segments; they attend over the whole sequence of HuBERT's final-layer features and return a fixed set of vectors that the frozen LLM treats as text-like inputs. Three alignment objectives during pre-training teach the adapter to bridge modalities: speech-text contrastive learning maximizes mutual information between paired speech and text, speech-text matching is a binary classification of whether an audio-text pair is aligned, and speech-text generation trains the adapter to generate text from audio autoregressively. Fine-tuning then optimizes the adapter alone with the LLM's cross-entropy objective, so the machinery's job is to distill language-neutral semantic content from acoustic features into queries the LLM can translate.

What would settle it

Run SparQLe on the MuST-C English-German tst-COMMON set and score the outputs with COMET-22 or human ratings instead of BERTScore; if the zero-shot score drops to the weak baseline or below, the claimed generalization is an artifact of the metric rather than real semantic translation.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that a Q-Former with 100 learnable query tokens is enough to align frozen HuBERT final-layer speech features to Llama 3 for automatic speech translation. SparQLe is claimed to be the first such system that uses only self-supervised speech representations and does not rely on a large supervised ASR encoder such as Whisper; both the speech encoder and the LLM remain frozen throughout. With 70% of fine-tuning data used for English recognition and 30% for English-French translation, the adapter learns to extract meaning that the LLM can render in French and, zero-shot, German, achieving BERTScore 85.56 on English-French and 83.26 on English-German on the MuST-C tst-COMMON set, compared with strong-baseline scores of 81.75 and 77.44. The paper also shows one example where a single prompt produces both a transcription and a translation, illustrating potential multi-task behavior, though it explicitly notes that this capability has not been formally evaluated.

Load-bearing premise

The system assumes that HuBERT's final-layer features, learned entirely from unlabeled English audio, contain enough language-neutral meaning that a Q-Former trained on English and French can map them into Llama 3's text space and produce correct translations into languages like German that the adapter never saw.

Editorial extensions

If this is right

  • If the reported results hold, speech translation to a new target language reduces to changing the language name in the instruction prompt; no new training for that language is needed.
  • The same frozen speech encoder and LLM, with only the Q-Former adapter trained, can handle both transcription and translation in one response, suggesting a single adapter can serve multiple speech-to-text tasks.
  • Because no large supervised ASR encoder is used, the approach lowers the cost of connecting speech to an LLM to training one small adapter on a single GPU for a few weeks.
  • The zero-shot English-German result implies the multilingual translation capacity lives mostly in the frozen LLM, with the adapter's role limited to preserving speech semantics rather than translating.
  • The architecture is not translation-specific, so the same trained adapter could in principle be applied to other speech understanding tasks such as summarization or question answering, provided the appropriate instruction prompts are used.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A stricter test the paper does not run is replacing BERTScore with a metric such as COMET or chrF on the same MuST-C English-German split; if the zero-shot gap shrinks or disappears, the semantic-translation claim would lose support because BERTScore rewards word overlap even when meaning differs.
  • The design implies that adding a small amount of non-English audio during the English pre-training stage might improve zero-shot robustness for structurally distant languages, and this could be tested by varying the pre-training language mix while keeping the fine-tuning pipeline fixed.
  • One could separate the contributions of the speech encoder and the LLM by swapping Llama 3 for a weaker multilingual LLM; if English-German scores collapse under that swap, the zero-shot transfer is inherited from the LLM's language ability rather than from the speech features.
  • The paper's single multi-task example suggests the same adapter could handle speech question answering or summarization by changing the prompt alone, and an immediate test would be running the released checkpoint on a speech question-answering benchmark.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes SparQLe, a parameter-efficient modality adapter that connects frozen HuBERT self-supervised speech representations to a frozen instruction-tuned Llama-3 model via a Q-Former with learnable query tokens. The adapter is pre-trained on English audio with ASR-style objectives and then fine-tuned on English recognition and English-to-French translation data. The authors report BERTScore results on MuST-C showing that SparQLe outperforms the IWSLT 2022 strong and weak baselines for English-to-French and, zero-shot, for English-to-German. The paper also presents qualitative examples of zero-shot translation into several languages and a multitask example, and it releases model code. The central empirical claim is the Table 2 BERTScore advantage, supported primarily by a single evaluation table and illustrative outputs.

Significance. If the Table 2 results are reliable, the paper makes a useful contribution: it demonstrates that a relatively simple Q-Former adapter, trained only on English and English-to-French data, can map frozen HuBERT features into a frozen LLM's embedding space and produce competitive speech translation, including into an unseen target language. The claim of being the first system to use frozen SSL speech representations without a Whisper-based ASR encoder is clearly supported by the related-work comparison in Table 1. The authors also release their code and models, which is a concrete strength that aids reproducibility. However, the significance of the headline result is currently limited by evaluation choices that may inflate the reported scores: prompt selection on an unidentified held-out set, unspecified post-hoc output cleaning, lack of statistical significance testing, and reliance on BERTScore as the sole metric. The conceptual approach is not invalidated by these issues, but the empirical evidence for the central claim needs to be made rigorous before the result can be accepted.

major comments (4)
  1. [§4.4.1, Table 2, and footnote 7] The reported BERTScore gains in Table 2 are not reliable as reported because the inference prompt was selected by taking the best of the four prompts in Section 4.4 on an unidentified held-out set, per footnote 7. If that held-out set is tst-COMMON or overlaps with the test set, then Table 2 reports a maximum over four prompt conditions on the test set rather than the performance of a fixed, pre-specified prompt. The paper must identify the held-out set, report per-prompt scores on both English-French and English-German, and state explicitly whether any statistic computed on the test split influenced the choice of prompt. Without this information, the magnitude of the 3.81-5.82 point gaps over the strong baseline cannot be assessed.
  2. [§4.4.1] The post-hoc cleanup of chat artifacts is described only as an 'endeavor to eliminate instances of prompt recurrence (chat artifacts) in the final text'; no concrete rules, no examples of removed artifacts, and no raw uncorrected scores are provided. If the cleanup rules were developed or tuned on the same test outputs, the comparison against the IWSLT baselines is biased in favor of SparQLe. The authors should specify a deterministic cleanup procedure, apply it identically to the baseline outputs, and report both raw and cleaned BERTScore values so the effect of the cleanup is visible.
  3. [§4.4.1 and Limitations] BERTScore is the only automatic metric used for the central comparison, and the Limitations section itself concedes that BERTScore assigns 99.7% F1 to 'never had any act seemed so impossible' versus 'always had any act seemed so impossible', which have opposite meaning. The zero-shot German output in Figure 3, 'jich möchte heute über energie und klima sprechen', contains a spelling error and word-order deviation from the reference, yet it would receive high lexical-overlap credit from BERTScore. The paper should report complementary metrics such as chrF, COMET, or human adequacy judgments, and should include bootstrap confidence intervals or significance tests for the differences in Table 2. Otherwise the claim that SparQLe 'preserves the semantic content' and 'outperforms' the baselines is not adequately supported.
  4. [§4.4.1 and §4.1] The manuscript does not state whether the IWSLT strong and weak baseline outputs were subjected to the same text normalization (lowercasing, punctuation removal, action-description removal) and to the same artifact-removal procedure before BERTScore was computed. If the baselines were scored with different preprocessing, the differences in Table 2 could reflect preprocessing rather than translation quality. The authors should specify the exact preprocessing and cleanup pipeline applied to each system's outputs, including the baselines.
minor comments (4)
  1. [Figure 3] The zero-shot examples contain typos and grammatical errors, e.g., German 'jich' for 'ich', Russian 'о энергии' instead of 'об энергии', and the Arabic lines appear incomplete or duplicated; these should be corrected or explicitly acknowledged as unedited model outputs, since the figure is used as qualitative evidence of multilingual zero-shot ability.
  2. [Section 4.4] The fourth prompt, 'Give me the Language translation of this "Language".', appears to be a typo; if the prompt is literally as written, it would be confusing to the LLM and should be revised or clarified.
  3. [Section 3.1] The description of pre-training states that 'each output vector from the Q-Former is successively fed into a modality adapter to predict the next token', but the relationship between the Q-Former output and the adapter's autoregressive prediction is not explained in enough detail to be reproduced; a diagram or explicit tensor-shape description would help.
  4. [Section 4.3.1] The phrase '457 × 2 hours of audio samples from MuST-C' is ambiguous; please clarify whether this means two languages at 457 hours each, or 457 hours total for two tasks, and specify the train/dev/test splits used.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the central comparison is an empirical train/eval loop on held-out splits; prompt selection on a held-out set is standard model selection, not equation-level circularity.

full rationale

SparQLe's derivation chain is empirical: a Q-Former adapter is pre-trained with English ASR-style objectives, fine-tuned on English plus English-French MuST-C data, and then evaluated on tst-COMMON against IWSLT 2022 baselines (Table 2). Nothing in this loop defines the claimed output in terms of itself: the adapter parameters are optimized with cross-entropy and contrastive losses on training splits, and the German result is a genuinely unseen target language not present in fine-tuning. The only evaluation-design issue is footnote 7, which reports selecting the best of four prompts on an unspecified held-out set; this is a model-selection step rather than a fitted-parameter-renamed-as-prediction, and without evidence that the held-out set is tst-COMMON it cannot be called circular. The BERTScore limitation acknowledged in the Limitations section ('never had any act seemed so impossible' vs 'always had any act seemed so impossible' scoring 99.7% F1) is a metric-validity caveat, not a case of the paper's conclusion being equivalent to its input metric. All external references (BLIP-2 Q-Former, HuBERT, Llama-3, SALMONN prompts, IWSLT baselines) are independent prior work, and there are no load-bearing self-citations. Therefore the paper's central claim retains independent empirical content.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim rests on a stack of domain assumptions: SSL feature sufficiency, query token efficacy, LLM cross-modal generation, and metric validity. No new entities are postulated; the approach reuses BLIP-2's Q-Former as a bridge.

free parameters (5)
  • Number of Q-Former query tokens = 100
    Chosen by hand in Section 4.2.1; no ablation is run on this value.
  • Fine-tuning data split (recognition vs translation) = 70% ASR, 30% En-Fr translation
    Deliberate split set in Section 4.3.1 to limit translation data; affects the balance of tasks.
  • Prompt template selection = Best of 4 prompts on held-out set
    Footnote 7: the best prompt was selected on the held-out set, a mild form of fitting to the evaluation.
  • Learning rate schedule endpoints = 1e-4 to 1e-5 with warmup 1e-6
    Set in Section 4.2.1; standard choice, not fitted.
  • Maximum audio length = 480K frames (30s)
    Set in Section 4.2.1; clips longer audio.
assumptions (5)
  • domain assumption HuBERT final-layer features carry sufficient semantic content for translation
    The model uses only HuBERT features and a Q-Former; if these features lack language-agnostic content, the adapter cannot recover it.
  • domain assumption The Q-Former with 100 query tokens can extract task-relevant information from the entire speech sequence
    Query tokens are placeholders with no fixed receptive field; the paper assumes they gather global information from the spoken utterance.
  • domain assumption A frozen instruction-tuned LLM can produce correct translations from continuous query embeddings
    The LLM is never fine-tuned; the paper assumes the adapter can inject speech-derived vectors that align with the LLM's text space.
  • domain assumption BERTScore is a valid evaluation metric for translation quality in this setting
    The authors rely on BERTScore as the only metric and even acknowledge its failure on negation examples in the Limitations section.
  • domain assumption IWSLT 2022 baselines are directly comparable to the SparQLe evaluation
    Table 2 reports BERTScore from the IWSLT challenge, but differences in training data, decoding, and artifact removal are not fully controlled.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SparQLe: Speech Queries to Text Translation Through LLMs." pith.science (2026). https://pith.science/paper/7I6NBOVB

@misc{pith2026250209284,
  author       = {Pith},
  title        = {Pith review of: SparQLe: Speech Queries to Text Translation Through LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7I6NBOVB}},
  note         = {Machine review of arXiv:2502.09284}
}
read the original abstract

With the growing influence of Large Language Models (LLMs), there is increasing interest in integrating speech representations with them to enable more seamless multi-modal processing and speech understanding. This study introduces a novel approach that combines self-supervised speech representations with instruction-tuned LLMs for speech-to-text translation. The proposed approach leverages a modality adapter to align extracted speech features with instruction-tuned LLMs using English speech data. Our experiments demonstrate that this method effectively preserves the semantic content of the input speech and serves as an effective bridge between self-supervised speech models and instruction-tuned LLMs, offering a promising approach for various speech understanding applications.

Figures

Figures reproduced from arXiv: 2502.09284 by the authors.

Figure 2
Figure 2. Modality adapter with auto-regressive super [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Sample of zero-shot instruction generation across multiple languages. To evaluate zero-shot capability, we [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Example from the SparQLe for multi-tasking [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

43 extracted references · 12 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    AI@Meta. 2024. https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md Llama 3 model card

    2024

  4. [4]

    Antonios Anastasopoulos, Loc Barrault, Luisa Bentivogli, Marcely Zanon Boito, Ond r ej Bojar, Roldano Cattoni, Anna Currey, Georgiana Dinu, Kevin Duh, Maha Elbayad, and 1 others. 2022. Findings of the iwslt 2022 evaluation campaign. In Proceedings of the 19th International Conference on Spoken Language Translation (IWSLT 2022), pages 98--157. Association ...

    work page 2022

  5. [5]

    Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33:12449--12460

    2020

  6. [6]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, and 29 others. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609

    arXiv 2023

  7. [7]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901

    2020

  8. [8]

    Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, and 1 others. 2022. Wavlm: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 16(6):1505--1518

    2022

Show all 43 references
  1. [9]

    Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Daniel Tompkins, Zhuo Chen, Wanxiang Che, Xiangzhan Yu, and Furu Wei. 2023. https://proceedings.mlr.press/v202/chen23ag.html BEAT s: Audio pre-training with acoustic tokenizers . In Proceedings of the 40th International Conference...

    2023

  2. [10]

    Zhehuai Chen, He Huang, Andrei Andrusenko, Oleksii Hrinchuk, Krishna C Puvvada, Jason Li, Subhankar Ghosh, Jagadeesh Balam, and Boris Ginsburg. 2024. Salm: Speech-augmented language model with in-context learning for speech recognition and translation. In ICASSP 2024-2024 IEEE...

    2024

  3. [11]

    Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, Chang Zhou, and Jingren Zhou. 2024. Qwen2-audio technical report. arXiv preprint arXiv:2407.10759

    2024 arXiv

  4. [12]

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinso...

    2024

  5. [13]

    Nilaksh Das, Saket Dingliwal, Srikanth Ronanki, Rohit Paturi, David Huang, Prashant Mathur, Jie Yuan, Dhanush Bekal, Xing Niu, Sai Muralidhar Jayanthi, and 1 others. 2024. Speechverse: A large-scale generalizable audio language model. arXiv preprint arXiv:2405.08295

    2024 arXiv

  6. [14]

    Di Gangi, Roldano Cattoni, Luisa Bentivogli, Matteo Negri, and Marco Turchi

    Mattia A. Di Gangi, Roldano Cattoni, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2019. https://doi.org/10.18653/v1/N19-1202 M u ST - C : a M ultilingual S peech T ranslation C orpus . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association...

    2019 doi

  7. [15]

    Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. Advances in neural information processing systems, 32

    2019

  8. [16]

    Qingkai Fang, Shoutao Guo, Yan Zhou, Zhengrui Ma, Shaolei Zhang, and Yang Feng. 2024. Llama-omni: Seamless speech interaction with large language models. arXiv preprint arXiv:2409.06666

    2024 arXiv

  9. [17]

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pages 2790--2799. PMLR

    2019

  10. [18]

    Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processin...

    2021

  11. [19]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685

    2021 arXiv

  12. [20]

    Hirofumi Inaguma, Shun Kiyono, Kevin Duh, Shigeki Karita, Nelson Yalta, Tomoki Hayashi, and Shinji Watanabe. 2020. Espnet-st: All-in-one speech translation toolkit. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstration...

    2020

  13. [21]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  14. [22]

    Surafel M Lakew, Marcello Federico, Yue Wang, Cuong Hoang, Yogesh Virkar, Roberto Barra-Chicote, and Robert Enyedi. 2021. Machine translation verbosity control for automatic dubbing. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (...

    2021

  15. [23]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730--19742. PMLR

    2023

  16. [24]

    Ziyang Ma, Guanrou Yang, Yifan Yang, Zhifu Gao, Jiaming Wang, Zhihao Du, Fan Yu, Qian Chen, Siqi Zheng, Shiliang Zhang, and Xie Chen. 2024. https://arxiv.org/abs/2402.08846 An embarrassingly simple approach for llm with strong asr capacity . Preprint, arXiv:2402.08846

    2024 arXiv

  17. [25]

    Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5206--5210. IEEE

    2015

  18. [26]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, and 1 others. 2021. Learning transferable visual models from natural language supervision. In International conference on machine...

    2021

  19. [27]

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning, pages 28492--28518. PMLR

    2023

  20. [28]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, and 1 others. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9

    2019

  21. [29]

    Fastspeech 2: Fast and high-quality end-to-end text to speech

    Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. Fastspeech 2: Fast and high-quality end-to-end text to speech. In International Conference on Learning Representations

  22. [30]

    Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun MA, and Chao Zhang. 2024. https://openreview.net/forum?id=14rn7HpKVk SALMONN : Towards generic hearing abilities for large language models . In The Twelfth International Conference on Learning ...

    2024

  23. [31]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, and 1 others. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

    2023 arXiv

  24. [32]

    Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. 2021. Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems, 34:200--212

    2021

  25. [33]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30

    2017

  26. [34]

    Chen Wang, Minpeng Liao, Zhongqiang Huang, Jinliang Lu, Junhong Wu, Yuchen Liu, Chengqing Zong, and Jiajun Zhang. 2023 a . Blsp: Bootstrapping language-speech pre-training via behavior alignment of continuation writing. arXiv preprint arXiv:2309.00916

    2023 arXiv

  27. [35]

    Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, and 1 others. 2023 b . Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301.02111

    2023 arXiv

  28. [36]

    Mingqiu Wang, Wei Han, Izhak Shafran, Zelin Wu, Chung-Cheng Chiu, Yuan Cao, Nanxin Chen, Yu Zhang, Hagen Soltau, Paul K Rubenstein, and 1 others. 2023 c . Slm: Bridge the thin gap between speech and text foundation models. In 2023 IEEE Automatic Speech Recognition and Understa...

    2023

  29. [37]

    Mingqiu Wang, Izhak Shafran, Hagen Soltau, Wei Han, Yuan Cao, Dian Yu, and Laurent El Shafey. 2023 d . Speech-to-text adapter and speech-to-entity retriever augmented llms for speech understanding. arXiv preprint arXiv:2306.07944

    2023 arXiv

  30. [38]

    Yongqiang Wang, Abdelrahman Mohamed, Due Le, Chunxi Liu, Alex Xiao, Jay Mahadeokar, Hongzhao Huang, Andros Tjandra, Xiaohui Zhang, Frank Zhang, and 1 others. 2020. Transformer-based acoustic modeling for hybrid speech recognition. In ICASSP 2020-2020 IEEE International Confere...

    2020

  31. [39]

    Qiantong Xu, Alexei Baevski, Tatiana Likhomanenko, Paden Tomasello, Alexis Conneau, Ronan Collobert, Gabriel Synnaeve, and Michael Auli. 2021. Self-training and pre-training are complementary for speech recognition. In ICASSP 2021-2021 IEEE International Conference on Acoustic...

    2021

  32. [40]

    Wenyi Yu, Changli Tang, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. 2024. Connecting speech encoder and large language model for asr. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), page...

    2024

  33. [41]

    Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. 2021. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495--507

    2021

  34. [42]

    Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. 2023. Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 15757--15773

    2023

  35. [43]

    Weinberger, and Yoav Artzi

    Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. https://openreview.net/forum?id=SkeHuCVFDr Bertscore: Evaluating text generation with bert . In International Conference on Learning Representations

    2020

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.