Pith. sign in

REVIEW 4 major objections 6 minor 11 cited by

MERaLiON-AudioLLM: Bridging Audio and Language with Large Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read MERaLiON-AudioLLM is claimed to be the first speech-text model built for Singapore's multilingual speech, and it reports lower local word-error rates plus gains in spoken question answering and summarization.

desk verdict Useful localized speech-text model and new corpus, but the evaluation table is too weak to back the abstract's task-understanding claim; okay as a resource report. read the letter →

arxiv 2412.09818 v3 pith:DTJUJODM submitted 2024-12-13 cs.CL cs.AI

Yingxu He , Zhuohan Liu , Shuo Sun , Bin Wang , Wenyu Zhang , Xunlong Zou , Nancy F. Chen , Ai Ti Aw This is my paper · ORCID
classification cs.CLcs.AI
keywords audiolargelanguagemodelspeech-textSingaporeEnglishspeechrecognitiontranslationspokenquestionansweringdialoguesummarizationmultimodalinstructiontuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This technical report introduces MERaLiON-AudioLLM, an end-to-end model that takes audio plus a text instruction and produces a text answer, designed for Singapore's multilingual and code-switched speech. The authors claim it is the first speech-text model tailored to this landscape, and that fusing a locally fine-tuned speech encoder with a Southeast Asian language model improves both speech recognition and task-specific understanding compared with general-purpose audio LLMs. If the claim holds, communities with distinctive accents and low-resource languages gain a locally usable alternative to cascaded speech-recognition-plus-LLM pipelines, where recognition errors can propagate into the final answer. The report backs the claim with a curated 260,000-hour training corpus, a public release of weights and a multitask subset of Singapore's National Speech Corpus, and evaluations against several published audio LLMs.

What carries the argument

The load-bearing mechanism is the MLP-100 adaptor, a two-layer MLP that reshapes the audio encoder's 1,500 frame embeddings, each of dimension 1,280, into 100 tokens of dimension 3,854 to match the text decoder's embedding size. This adaptor, together with the fully fine-tuned MERaLiON-Whisper encoder, is trained end-to-end with the SEA-LION V3 decoder, whose MLP layers receive rank-8 LoRA adapters rather than full fine-tuning. The training objective is standard autoregressive cross-entropy over the text output conditioned on both audio and text instruction tokens, and the data pipeline filters the National Speech Corpus for mislabels, assigns identical transcriptions to the same splits, superimposes two-sided conversations, and segments audio to at most 30 seconds.

What would settle it

Select a random sample of responses from the spoken question answering, dialogue summarization, and speech instruction test sets and have human annotators score them with the same rubric; if the judge's rankings do not match human rankings, or if a different judge or prompt reverses the model ordering, the task-understanding claim fails. For the speech recognition part, resample the National Speech Corpus test items to form confidence intervals around the word error rate; the localization claim would be weakened if the 5% versus 19% margin over Qwen2-Audio on MNSC-ASR-Part 2 is within sampling noise.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is that localization transfers through the whole pipeline: fine-tuning the Whisper-large-v2 encoder on cleaned Singapore English data and connecting it end-to-end to a locally pre-trained LLM yields word error rates of 5% on the Multitask National Speech Corpus prompted readings, versus 19% for Qwen2-Audio and 33% for Whisper-large-v2, while the same model stays competitive on unseen Earnings21 and Earnings22 and improves on spoken question answering, dialogue summarization, speech instruction, and accent and gender recognition relative to the audio-LLM baselines. The improvement is attributed chiefly to data curation and in-domain fine-tuning rather than to new architecture, since the fusion design follows other audio LLMs. The paper also reports that the model underperforms on MELD sentiment and emotion, and flags instruction-following loss and a 30-second audio context limit as known limitations.

Load-bearing premise

The task-understanding part of the claim rests on automated judge scores whose reliability is never established, so if the judge is noisy or biased, the reported gains in spoken QA, summarization, instruction, and paralinguistic tasks are unsupported.

Editorial extensions

If this is right

  • A model fine-tuned on local data can beat general-purpose audio LLMs on in-country benchmarks while remaining competitive on standard ones such as LibriSpeech and Common Voice.
  • End-to-end audio-plus-text fusion gives a single model for speech recognition, translation, spoken question answering, dialogue summarization, and speech instruction, avoiding the error propagation of a separate ASR system feeding an LLM.
  • Publishing the model weights and the Multitask National Speech Corpus lets other groups reproduce the local-curation recipe and apply it to their own regional languages and accents.
  • The reported gaps on MELD sentiment and emotion indicate that paralinguistic understanding is not solved by this release and needs additional local data or architectural changes, as the paper itself states.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: if the automated judge scores are later validated by human agreement, the same recipe of fine-tuning a strong open encoder on a national corpus and fusing it with a locally pre-trained LLM should transfer to other under-resourced languages and dialects, because neither component is Singapore-specific.
  • A testable extension the paper does not run: measure word error rate on Singaporean conversational or telephone audio recorded after the training corpus was collected, to separate genuine local robustness from in-domain memorization of National Speech Corpus prompts.
  • An editor's inference: the 30-second limit and reported instruction-following loss suggest the release is a proof of concept; a concrete follow-up is to check whether replaying text-only instruction pairs during multimodal fine-tuning restores instruction following without eroding the speech recognition gains.
  • With the corpus released, an independent audit could verify that no identical transcription appears in both training and test splits, which would directly test the paper's data-leakage-avoidance claim.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This technical report introduces MERaLiON-AudioLLM, a speech-text model that combines a Whisper-large-v2-based encoder fine-tuned on Singaporean and Southeast Asian speech data with the SEA-LION V3 Gemma-2-based decoder, connected by an MLP-100 adaptor. The authors release model weights and a multitask corpus (MNSC), and evaluate the model on ASR, speech translation, spoken QA, dialogue summarization, speech instruction following, and paralinguistics against Qwen2-Audio, WavLLM, SALMONN, and a cascaded Whisper+SEA-LION baseline. The paper claims improvements in both speech recognition and task-specific understanding, and positions the model as the first speech-text model tailored for Singapore's multilingual landscape. The evidence for the task-specific claim rests on LLM-as-a-Judge scores in Table 1, while the ASR gains are mostly on NSC-derived test sets that overlap with the training corpus.

Significance. If the empirical claims survive scrutiny, the contribution would be a useful localized speech-text model and a reusable multitask corpus for Singapore English and regional languages, with publicly released weights that could support downstream work in low-resource and dialect-rich settings. The engineering details (data pipeline, FSDP training, compute infrastructure) are also valuable for practitioners. However, the central claims as written are not yet independently established: the task-specific understanding results are based on an unvalidated judge, the comparison to the cascaded baseline contradicts part of the abstract, and the strongest ASR gains are substantially in-domain. The released assets are a genuine community contribution, but the paper's evaluation must be strengthened and its claims scoped before the contribution can be assessed fairly.

major comments (4)
  1. [Section 5.1 and Table 1] The abstract claims 'improvements in task-specific understanding,' but the only evidence for the SQA, SDS, SI, and paralinguistic rows is the LLM-as-a-Judge framework described in Section 5.1. The manuscript does not name the judge model, give the prompt template, report human agreement or calibration, or provide confidence intervals. On this evidence, differences such as SLUE-SQA-5 82.9 vs 80.1 or MNSC-SQA-Part 3 51.4 vs 42.0 cannot be distinguished from judge noise or prompt sensitivity. This issue is load-bearing for the central claim and needs either validated judge scores with human agreement and intervals, or a re-scoped claim that does not assert general task-specific improvements.
  2. [Table 1 and Abstract] The phrase 'improvements in both speech recognition and task-specific understanding' is ambiguous and, on the table as reported, partly contradicted. The cascaded Whisper-large-v2 + SEA-LION baseline beats MERaLiON-AudioLLM on 5 of 8 SQA rows, 3 of 4 SDS rows, both SI rows, both MELD rows, and the two Earnings ASR rows (Earnings21-Test 0.17 vs 0.11; Earnings22-Test 0.20 vs 0.14). If the intended comparison is against other AudioLLMs, the abstract should say so explicitly; if it is against the cascaded pipeline, the table does not support the sentence. The claim needs to be restated with an explicit comparison class and the contradictions addressed.
  3. [Section 3 and Table 1 (MNSC rows)] The NSC-derived test sets come from the same corpus used for training the audio encoder and for multimodal instruction fine-tuning. The large MNSC-ASR-Part-2 gain (0.05 vs 0.19 for Qwen2-Audio) therefore largely demonstrates fit to the training distribution, as the paper itself concedes in Section 5.2 ('given its training on in-domain data'). Because localization is a central contribution, the evaluation needs external held-out local data, or a clear statement that the NSC results are in-domain checks rather than evidence of generalization. The conclusion in Section 8 should be tempered accordingly.
  4. [Table 1 (overall)] All results are single point estimates with no error bars, significance tests, or multiple-seed variation. For rows where margins are small (e.g., LibriSpeech-Test-Clean 0.03 vs 0.03; CoVoST 2 Zh→En 15.0 vs 16.5), the reported 'competitive' or 'best' rankings are not established. Because the central claim depends on relative performance, the paper should report variability (e.g., confidence intervals or multiple runs) or soften comparative statements.
minor comments (6)
  1. [Section 1 and footnote 3] The Hugging Face link is given as 'MERaLiON/AudioLLM' in Section 1 but as 'MERaLiON/MERaLiON-AudioLLM-Whisper-SEA-LION' in the footnote; please unify the URLs.
  2. [Section 5.2] In the paragraph on MELD, 'the goal is to identity the sentiment' should be 'identify the sentiment'; in Section 7, 'we intent to explore' should be 'we intend to explore'.
  3. [Section 5.1] For the cascaded baseline, only 'We tuned its hyperparameters and prompt template' is reported; please provide the prompt template and hyperparameter choices so the comparison is reproducible.
  4. [Section 3] The MNSC corpus is released, but its processing details (segmentation, deduplication, split assignment, and the synthesis of SQA/SDS/GR data) are deferred to future work; a data card or appendix with these details would strengthen the release's reproducibility.
  5. [Table 1] The underlining and bolding conventions are not consistently visible when the cascaded column contains the best result (e.g., several rows have the best score in the Cascaded Model column but are not underlined in the text version); please clarify the formatting.
  6. [Section 2.1.2] The MLP-100 adaptor is claimed to give 'slightly better results' than window-level Qformer and ConvMLP, but no comparison numbers or ablations are shown; please add the supporting experiment or remove the claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the evaluation table and the disclosed in-domain NSC results support an empirical system report, not a derivation that reduces to its inputs.

full rationale

MERaLiON-AudioLLM is presented as an empirical system description rather than a first-principles derivation. The claimed chain is: initialize from Whisper-large-v2 and SEA-LION V3, fine-tune on NSC-derived and synthesized instruction data, then evaluate on AudioBench test sets. The NSC/MNSC results are in-domain, and the paper explicitly acknowledges this: “As expected, AudioLLM performs better on the NSC datasets, given its training on in-domain data.” That disclosure makes the in-domain numbers a documented benchmark comparison rather than a disguised prediction: the test splits are held out, the training loss in Eq. (1) is not minimized on those test labels, and no parameter fitted to a test subset is renamed as a prediction. External evaluations on LibriSpeech, Common Voice, Earnings21/22, and CoVoST 2 provide independent grounding for the core speech-recognition comparison. The LLM-as-a-Judge protocol is delegated to Wang et al. 2024, a self-citation, but it is a methodological pointer rather than a load-bearing uniqueness theorem or an ansatz that forces the conclusions; the unverified reliability of judge scores is a validity concern, not a circularity. No displayed equation or construction makes a claimed output equal to an input, and no central claim is defined in terms of the quantity it is supposed to predict. Therefore no self-definitional, fitted-input, or self-citation reduction is present.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

No new theoretical entities are posited. The released model and MNSC corpus are engineering artifacts, not postulated constructs. The central claim rests mainly on the reliability of the evaluation protocol and the representativeness of NSC/MNSC data.

free parameters (4)
  • MLP-100 compression factor s = 15
    Hand-chosen to reduce Whisper's 1500-length embeddings to 100 tokens (1500/15). No ablation is reported against other s values.
  • LoRA rank = 8
    Chosen for decoder fine-tuning; no comparison with other ranks is reported.
  • Peak learning rate = 5e-5
    Standard fine-tuning choice; no sensitivity analysis is provided.
  • Training steps and batch size = 200k steps, global batch 640
    This is the compute budget used for the release; no convergence analysis or ablation is given.
assumptions (3)
  • domain assumption LLM-as-a-Judge scores are a valid measure of task-specific understanding
    Used for SQA, SDS, SI, and paralinguistic tasks without specifying the judge model, reporting human agreement, or providing error bars (Section 5.1).
  • domain assumption NSC/MNSC test splits are representative of Singapore English and free of leakage
    The localized central claim relies on NSC-derived evaluation sets; the paper describes filtering and split consistency for Parts 1 and 2, but cannot prove generalizability (Sections 3 and 5.2).
  • domain assumption Whisper-large-v2 features fine-tuned on NSC are sufficient audio representations for Singlish and code-switched speech
    This architecture choice is made without comparing other encoders in this report (Section 2.1.1).

how reviews work

0 comments
Cite this review

Pith. "Pith review of MERaLiON-AudioLLM: Bridging Audio and Language with Large Language Models." pith.science (2026). https://pith.science/paper/DTJUJODM

@misc{pith2026241209818,
  author       = {Pith},
  title        = {Pith review of: MERaLiON-AudioLLM: Bridging Audio and Language with Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DTJUJODM}},
  note         = {Machine review of arXiv:2412.09818}
}
read the original abstract

We introduce MERaLiON-AudioLLM (Multimodal Empathetic Reasoning and Learning in One Network), the first speech-text model tailored for Singapore's multilingual and multicultural landscape. Developed under the National Large Language Models Funding Initiative, Singapore, MERaLiON-AudioLLM integrates advanced speech and text processing to address the diverse linguistic nuances of local accents and dialects, enhancing accessibility and usability in complex, multilingual environments. Our results demonstrate improvements in both speech recognition and task-specific understanding, positioning MERaLiON-AudioLLM as a pioneering solution for region specific AI applications. We envision this release to set a precedent for future models designed to address localised linguistic and cultural contexts in a global framework.

Figures

Figures reproduced from arXiv: 2412.09818 by the authors.

Figure 1
Figure 1. Architecture of MERaLiON-AudioLLM: MERaLiON-AudioLLM fuses MERaLiON￾Whisper with AI Singapore’s SEA-LION V3. and contextual nuances is crucial for creating more inclusive and effective AI systems. Traditional speech recognition models often struggle with the diversity of accents, dialects, and linguistic subtleties, leading to inaccuracies and reduced usability in complex, multilingual environments. By addressing th… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Multilingual Emotion Neurons in Large Audio-Language Models

    cs.CL 2026-08 conditional novelty 7.0 of 10

    Emotion neurons in audio-language models are mostly language-specific, but pooling evidence across languages finds a shared set whose causal manipulation transfers to unseen languages.

  2. Towards Spoken Mathematical Reasoning: Benchmarking Speech-based Models over Multi-faceted Math Problems

    cs.CL 2025-05 conditional novelty 7.0 of 10

    A new spoken math benchmark, Spoken-MQA, shows that current speech-based AI models reason poorly from spoken math input, especially for arithmetic and knowledge-heavy problems.

  3. AsyncSwitch: Asynchronous Text-Speech Adaptation for Code-Switched ASR

    cs.CL 2025-06 conditional novelty 6.0 of 10

    AsyncSwitch improves code-switched ASR on Whisper by adapting the decoder on text before speech-text alignment and full fine-tuning.

  4. Bi-directional Context-Enhanced Speech Large Language Models for Multilingual Conversational ASR

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Adding language-specific prompts and bi-directional conversational context to a speech LLM cuts validation error by 18% relative and edges out a model trained on four times more data.

  5. Can Quantized Audio Language Models Perform Zero-Shot Spoofing Detection?

    cs.SD 2025-06 conditional novelty 6.0 of 10

    Zero-shot audio-language models are not reliable spoof detectors because they over-predict 'spoof', and FP16 quantization keeps this bias while INT8 worsens it.

  6. Advancing Singlish Understanding: Bridging the Gap with Datasets and Multimodal Models

    cs.CL 2025-01 conditional novelty 6.0 of 10

    The authors release MNSC, the largest standardized multitask spoken Singlish corpus, and SingAudioLLM, a multimodal model that sets strong baselines on ASR, spoken QA, dialogue summarization, and paralinguistic QA.

  7. Incorporating Contextual Paralinguistic Understanding in Large Speech-Language Models

    cs.CL 2025-08 conditional novelty 5.0 of 10

    Training a speech-LLM on question-answer pairs generated with both discrete and continuous emotion labels improves its contextual emotion reasoning as scored by an LLM judge.

  8. Hidden in Plain Sight: Evaluation of the Deception Detection Capabilities of LLMs in Multimodal Settings

    cs.CL 2025-06 conditional novelty 5.0 of 10

    An evaluation of 7 LLMs/LMMs on 3 deception datasets shows fine-tuned text LLMs set benchmarks on review spam while multimodal models lag behind video-based baselines.

  9. Contextual Paralinguistic Data Creation for Multi-Modal Speech-LLM: Data Condensation and Spoken QA Generation

    cs.CL 2025-05 conditional novelty 5.0 of 10

    A pipeline condenses in-the-wild speech with emotion labels and uses ChatGPT to generate contextual paralinguistic QA pairs, released as a 480-sample benchmark.

  10. Qwen vs. Gemma Integration with Whisper: A Comparative Study in Multilingual SpeechLLM Systems

    cs.CL 2025-06 conditional novelty 4.0 of 10

    A Whisper-large-v3 encoder with a linear projector and LoRA-tuned Gemma3-12B decoder achieves 16.63% average WER/CER on the MLC-SLM 2025 private test set.

  11. MERaLiON-SpeechEncoder: Towards a Speech Foundation Model for Singapore and Beyond

    cs.CL 2024-12 conditional novelty 4.0 of 10

    An openly released 630M-parameter BEST-RQ speech encoder, pretrained on 200k hours, matches state-of-the-art encoders on several ASR benchmarks and holds its own on ten SUPERB tasks.

Reference graph

Works this paper leans on

67 extracted references · 57 canonical work pages · cited by 11 Pith papers

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    SEA-LION (southeast asian languages in one network): A family of large language models for southeast asia

    AI Singapore . SEA-LION (southeast asian languages in one network): A family of large language models for southeast asia. https://github.com/aisingapore/sealion, 2024

    work page 2024

  4. [4]

    Llama 3 model card, 2024

    AI@Meta. Llama 3 model card, 2024. URL https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md

    2024

  5. [5]

    K. An, Q. Chen, C. Deng, Z. Du, C. Gao, Z. Gao, Y. Gu, T. He, H. Hu, K. Hu, S. Ji, Y. Li, Z. Li, H. Lu, H. Luo, X. Lv, B. Ma, Z. Ma, C. Ni, C. Song, J. Shi, X. Shi, H. Wang, W. Wang, Y. Wang, Z. Xiao, Z. Yan, Y. Yang, B. Zhang, Q. Zhang, S. Zhang, N. Zhao, and S. Zheng. FunAudioLLM : Voice understanding and generation foundation models for natural interac...

    work page 2024

  6. [6]

    Ardila, M

    R. Ardila, M. Branson, K. Davis, M. Kohler, J. Meyer, M. Henretty, R. Morais, L. Saunders, F. Tyers, and G. Weber. Common Voice : A massively-multilingual speech corpus. In Language Resources and Evaluation Conference, pages 4218--4222, 2020

    work page 2020

  7. [7]

    Press release on Singapore's National Multimodal Large Language Model Programme , 2023

    A*STAR. Press release on Singapore's National Multimodal Large Language Model Programme , 2023. URL https://www.a-star.edu.sg/i2r/news-accolades/news-accolades/press-releases/NewsNAccolades/press-releases/SEA-LLM

    work page 2023

  8. [8]

    J. Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, and et al. Qwen technical report. arXiv, 2023

    work page 2023

Show all 67 references
  1. [9]

    K. Chen, Y. Gou, R. Huang, Z. Liu, D. Tan, J. Xu, C. Wang, Y. Zhu, Y. Zeng, K. Yang, D. Wang, K. Xiang, H. Li, H. Bai, J. Han, X. Li, W. Jin, N. Xie, Y. Zhang, J. T. Kwok, H. Zhao, X. Liang, D.-Y. Yeung, X. Chen, Z. Li, W. Zhang, Q. Liu, L. Hong, L. Hou, H. Xu, and H. Kong. EM...

    2024

  2. [10]

    S. Chen, C. Wang, Z. Chen, Y. Wu, S. Liu, and et al. WavLM : Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 16 0 (6): 0 1505--1518, 2022

    2022

  3. [11]

    S. Chen, Y. Wu, C. Wang, S. Liu, D. Tompkins, Z. Chen, W. Che, X. Yu, and F. Wei. BEATs : Audio pre-training with acoustic tokenizers. In International Conference on Machine Learning (ICML), 2023

    2023

  4. [12]

    X. Chen, S. Zhang, Q. Bai, K. Chen, and S. Nakamura. LLaST : Improved end-to-end speech translation system leveraged by large language models. In Annual Meeting of the Association for Computational Linguistics (ACL), 2024 b

    2024

  5. [13]

    Z. Chen, H. Huang, O. Hrinchuk, K. C. Puvvada, N. R. Koluguri, P. Żelasko, J. Balam, and B. Ginsburg. BESTOW : Efficient and streamable speech language model with the best of two worlds in GPT and T5 . arXiv, 2024 c

    2024

  6. [14]

    Y. Chu, J. Xu, X. Zhou, Q. Yang, S. Zhang, Z. Yan, C. Zhou, and J. Zhou. Qwen-Audio : Advancing universal audio understanding via unified large-scale audio-language models. arXiv, 2023

    2023

  7. [15]

    Y. Chu, J. Xu, Q. Yang, H. Wei, X. Wei, Z. Guo, Y. Leng, Y. Lv, J. He, J. Lin, C. Zhou, and J. Zhou. Qwen2-Audio technical report. arXiv, 2024

    2024

  8. [16]

    C. Cui, Y. Ma, X. Cao, W. Ye, Y. Zhou, K. Liang, J. Chen, J. Lu, Z. Yang, K.-D. Liao, T. Gao, E. Li, K. Tang, Z. Cao, T. Zhou, A. Liu, X. Yan, S. Mei, J. Cao, Z. Wang, and C. Zheng. A survey on multimodal large language models for autonomous driving. In IEEE/CVF Winter Confere...

    2024

  9. [17]

    Deshmukh, B

    S. Deshmukh, B. Elizalde, R. Singh, and H. Wang. Pengi : An audio language model for audio tasks. In Conference on Neural Information Processing Systems (NeurIPS), 2023

    2023

  10. [18]

    Défossez, L

    A. Défossez, L. Mazaré, M. Orsini, A. Royer, P. Pérez, H. Jégou, E. Grave, and N. Zeghidour. Moshi : a speech-text foundation model for real-time dialogue. arXiv, 2024

    2024

  11. [19]

    Elfwing, E

    S. Elfwing, E. Uchibe, and K. Doya. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural networks, 107: 0 3--11, 2018

    2018

  12. [20]

    Q. Fang, S. Guo, Y. Zhou, Z. Ma, S. Zhang, and Y. Feng. LLaMA-Omni : Seamless speech interaction with large language models. arXiv, 2024

    2024

  13. [21]

    Fathullah, C

    Y. Fathullah, C. Wu, E. Lakomkin, K. Li, J. Jia, Y. Shangguan, J. Mahadeokar, O. Kalinli, C. Fuegen, and M. Seltzer. AudioChatLlama : Towards general-purpose speech abilities for LLMs . In Conference of the North American Chapter of the Association for Computational Linguistic...

    2024

  14. [22]

    Ghosh, S

    S. Ghosh, S. Kumar, A. Seth, C. K. R. Evuru, U. Tyagi, S. Sakshi, O. Nieto, R. Duraiswami, and D. Manocha. GAMA : A large audio-language model with advanced audio understanding and complex reasoning abilities. arXiv, 2024

    2024

  15. [23]

    Y. Gong, A. H. Liu, H. Luo, L. Karlinsky, and J. Glass. Joint audio and speech understanding. In IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2023

    2023

  16. [24]

    Y. Gong, H. Luo, A. H. Liu, L. Karlinsky, and J. Glass. Listen, think, and understand. In International Conference on Learning Representations (ICLR), 2024

    2024

  17. [25]

    W. Held, E. Li, M. Ryan, W. Shi, Y. Zhang, and D. Yang. Distilling an end-to-end voice assistant without instruction training data. arXiv, 2024

    2024

  18. [26]

    Hendrycks and K

    D. Hendrycks and K. Gimpel. Gaussian error linear units (GELUs) . arXiv, 2023

    2023

  19. [27]

    W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed. HuBERT : Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29: 0 3451–3460, Oct 2021

    2021

  20. [28]

    E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. Lo RA : Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 2022

    2022

  21. [29]

    S. Hu, L. Zhou, S. Liu, S. Chen, L. Meng, H. Hao, J. Pan, X. Liu, J. Li, S. Sivasankaran, L. Liu, and F. Wei. WavLLM : Towards robust and adaptive speech large language model. arXiv, 2024

    2024

  22. [30]

    S. Ji, Y. Chen, M. Fang, J. Zuo, J. Lu, H. Wang, Z. Jiang, L. Zhou, S. Liu, X. Cheng, X. Yang, Z. Wang, Q. Yang, J. Li, Y. Jiang, J. He, Y. Chu, J. Xu, and Z. Zhao. WavChat : A survey of spoken dialogue models. arXiv, 2024

    2024

  23. [31]

    W. Kang, J. Jia, C. Wu, W. Zhou, E. Lakomkin, Y. Gaur, L. Sari, S. Kim, K. Li, J. Mahadeokar, and O. Kalinli. Frozen large language models can perceive paralinguistic aspects of speech. arXiv, 2024

    2024

  24. [32]

    J. X. Koh, A. Mislan, K. Khoo, B. Ang, W. Ang, C. Ng, and Y.-Y. Tan. Building the Singapore English National Speech Corpus . In Interspeech, pages 321--325, 2019

    2019

  25. [33]

    Z. Kong, A. Goel, R. Badlani, W. Ping, R. Valle, and B. Catanzaro. Audio Flamingo : A novel audio language model with few-shot learning and dialogue abilities. arXiv, 2024

    2024

  26. [34]

    Lee, S.-L

    C.-H. Lee, S.-L. Wu, C.-L. Liu, and H.-y. Lee. Spoken SQuAD : A study of mitigating the impact of speech recognition errors on listening comprehension. In Interspeech, 2018

    2018

  27. [35]

    J. Li, A. Hassani, S. Walton, and H. Shi. ConvMLP : Hierarchical convolutional MLPs for vision. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6307--6316, 2023

    2023

  28. [36]

    K.-H. Lu, Z. Chen, S.-W. Fu, H. Huang, B. Ginsburg, Y.-C. F. Wang, and H. yi Lee. DeSTA : Enhancing speech language models through descriptive speech-text alignment. arXiv, 2024

    2024

  29. [37]

    Z. Ma, G. Yang, Y. Yang, Z. Gao, J. Wang, Z. Du, F. Yu, Q. Chen, S. Zheng, S. Zhang, and X. Chen. An embarrassingly simple approach for llm with strong asr capacity. arXiv, 2024

    2024

  30. [38]

    Meralion-speechencoder: Towards a speech foundation model for singapore and beyond, 2024

    MERaLiON Team . Meralion-speechencoder: Towards a speech foundation model for singapore and beyond, 2024. URL https://arxiv.org/abs/2412.11538

    2024 arXiv

  31. [39]

    Minaee, T

    S. Minaee, T. Mikolov, N. Nikzad, M. A. Chenaghlu, R. Socher, X. Amatriain, and J. Gao. Large language models: A survey. arXiv, 2024

    2024

  32. [40]

    Nagrani, J

    A. Nagrani, J. S. Chung, and A. Zisserman. VoxCeleb : A large-scale speaker identification dataset. In Interspeech, 2017

    2017

  33. [41]

    T. A. Nguyen, B. Muller, B. Yu, M. R. Costa-jussa, M. Elbayad, S. Popuri, C. Ropers, P.-A. Duquenne, R. Algayres, R. Mavlyutov, I. Gat, M. Williamson, G. Synnaeve, J. Pino, B. Sagot, and E. Dupoux. Spirit LM : Interleaved spoken and written language model. Transactions of the ...

    2024

  34. [42]

    Panayotov, G

    V. Panayotov, G. Chen, D. Povey, and S. Khudanpur. Librispeech: An ASR corpus based on public domain audio books. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2015

    2015

  35. [43]

    Papineni, S

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. BLEU : a method for automatic evaluation of machine translation. In Annual Meeting of the Association for Computational Linguistics (ACL), 2002

    2002

  36. [44]

    D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le. SpecAugment : A simple data augmentation method for automatic speech recognition. In Interspeech, 2019

    2019

  37. [45]

    Paszke, S

    A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer. Automatic differentiation in pytorch. 2017

    2017

  38. [46]

    Poria, D

    S. Poria, D. Hazarika, N. Majumder, G. Naik, E. Cambria, and R. Mihalcea. MELD : A multimodal multi-party dataset for emotion recognition in conversations. In Annual Meeting of the Association for Computational Linguistics (ACL), 2019

    2019

  39. [47]

    Radford, J

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever. Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning (ICML), 2023

    2023

  40. [48]

    M. Rio, N. Delworth, R. Westerman, M. Huang, N. Bhandari, J. Palakapilly, Q. McNamara, J. Dong, P. Żelasko, and M. Jette. Earnings-21: A practical benchmark for ASR in the wild. In Interspeech, 2021

    2021

  41. [49]

    M. Rio, H. Peter, Q. McNamara, C. Miller, and S. Chandra. Earnings-22: A practical benchmark for accents in the wild. arXiv, 2022

    2022

  42. [50]

    P. K. Rubenstein, C. Asawaroengchai, D. D. Nguyen, A. Bapna, Z. Borsos, F. de Chaumont Quitry, P. Chen, D. E. Badawy, W. Han, E. Kharitonov, H. Muckenhirn, D. Padfield, J. Qin, D. Rozenberg, T. Sainath, J. Schalkwyk, M. Sharifi, M. T. Ramanovich, M. Tagliasacchi, A. Tudor, M. ...

    2023

  43. [51]

    S. Shon, S. Arora, C.-J. Lin, A. Pasad, F. Wu, R. S. Sharma, W.-L. Wu, H.-y. Lee, K. Livescu, and S. Watanabe. SLUE phase-2: A benchmark suite of diverse spoken language understanding tasks. In A. Rogers, J. Boyd-Graber, and N. Okazaki, editors, Proceedings of the 61st Annual ...

    2023 doi

  44. [52]

    C. Tang, W. Yu, G. Sun, X. Chen, T. Tan, W. Li, L. Lu, Z. Ma, and C. Zhang. SALMONN : Towards generic hearing abilities for large language models. International Conference on Learning Representations (ICLR), 2024

    2024

  45. [53]

    Taori, I

    R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto. Stanford Alpaca : An instruction-following LLaMA model. https://github.com/tatsu-lab/stanford_alpaca, 2023

    2023

  46. [54]

    G. Team, M. Riviere, S. Pathak, P. G. Sessa, C. Hardin, S. Bhupatiraju, L. Hussenot, T. Mesnard, B. Shahriari, A. Ramé, J. Ferret, P. Liu, P. Tafti, A. Friesen, M. Casbon, S. Ramos, R. Kumar, C. L. Lan, S. Jerome, A. Tsitsulin, N. Vieillard, P. Stanczyk, S. Girgin, N. Momchev,...

    2024

  47. [55]

    T. M. M. Team. streaming. <https://github.com/mosaicml/streaming/>, 2022

    2022

  48. [56]

    OpenHermes 2.5: An open dataset of synthetic data for generalist LLM assistants, 2023

    Teknium. OpenHermes 2.5: An open dataset of synthetic data for generalist LLM assistants, 2023. URL https://huggingface.co/datasets/teknium/OpenHermes-2.5

    2023

  49. [57]

    Touvron, L

    H. Touvron, L. Martin, K. R. Stone, P. Albert, A. Almahairi, and et al. Llama 2: Open foundation and fine-tuned chat models. arXiv, 2023

    2023

  50. [58]

    B. Wang, X. Zou, G. Lin, S. Sun, Z. Liu, W. Zhang, Z. Liu, A. Aw, and N. F. Chen. Audiobench: A universal benchmark for audio large language models. arXiv preprint arXiv:2406.16020, 2024

    2024 arXiv

  51. [59]

    B. Wang, X. Zou, S. Sun, W. Zhang, Y. He, Z. Liu, C. Wei, N. F. Chen, and A. Aw. Advancing singlish understanding: Bridging the gap with datasets and multimodal models. arXiv preprint arXiv:2501.01034, 2025

    2025 arXiv

  52. [60]

    C. Wang, A. Wu, J. Gu, and J. Pino. CoVoST 2 and massively multilingual speech translation. In Interspeech, pages 2247--2251, 2021

    2021

  53. [61]

    T. Wang, L. Zhou, Z. Zhang, Y. Wu, S. Liu, Y. Gaur, Z. Chen, J. Li, and F. Wei. VioLA : Unified codec language models for speech recognition, synthesis, and translation. arXiv, 2023

    2023

  54. [62]

    J. Wu, Y. Gaur, Z. Chen, L. Zhou, Y. Zhu, T. Wang, J. Li, S. Liu, B. Ren, L. Liu, and Y. Wu. On decoder-only architecture for speech-to-text and large language model integration. arXiv, 2023

    2023

  55. [63]

    Xie and C

    Z. Xie and C. Wu. Mini-Omni : Language models can hear, talk while thinking in streaming. arXiv, 2024

    2024

  56. [64]

    Xu, S.-X

    Y. Xu, S.-X. Zhang, J. Yu, Z. Wu, and D. Yu. Comparing discrete and continuous space LLMs for speech recognition. In Interspeech, 2024

    2024

  57. [65]

    W. Yu, C. Tang, G. Sun, X. Chen, T. Tan, W. Li, L. Lu, Z. Ma, and C. Zhang. Connecting speech encoder and large language model for ASR . In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2024

    2024

  58. [66]

    Zhang, S

    D. Zhang, S. Li, X. Zhang, J. Zhan, P. Wang, Y. Zhou, and X. Qiu. SpeechGPT : Empowering large language models with intrinsic cross-modal conversational abilities. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023

    2023

  59. [67]

    Zhang, S

    W. Zhang, S. Sun, B. Wang, X. Zou, Z. Liu, Y. He, G. Lin, N. F. Chen, and A. T. Aw. MoWE-Audio : Multitask audiollms with mixture of weak encoders. arXiv, 2024

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.