Adapting speech-aware LLMs with speaker cluster identification tags and concatenated multi-speaker data yields superior speaker-attributed ASR performance versus sequential diarization-plus-ASR pipelines.
hub
An embarrassingly simple approach for LLM with strong ASR capacity
19 Pith papers cite this work. Polarity classification is still indexing.
19
Pith papers citing it
abstract
In this paper, we focus on solving one of the most important tasks in the field of speech processing, i.e., automatic speech recognition (ASR), with speech foundation encoders and large language models (LLM). Recent works have complex designs such as compressing the output temporally for the speech encoder, tackling modal alignment for the projector, and utilizing parameter-efficient fine-tuning for the LLM. We found that delicate designs are not necessary, while an embarrassingly simple composition of off-the-shelf speech encoder, LLM, and the only trainable linear projector is competent for the ASR task. To be more specific, we benchmark and explore various combinations of LLMs and speech encoders, leading to the optimal LLM-based ASR system, which we call SLAM-ASR. The proposed SLAM-ASR provides a clean setup and little task-specific design, where only the linear projector is trained. To the best of our knowledge, SLAM-ASR achieves the best performance on the Librispeech benchmark among LLM-based ASR models and even outperforms the latest LLM-based audio-universal model trained on massive pair data. Finally, we explore the capability emergence of LLM-based ASR in the process of modal alignment. We hope that our study can facilitate the research on extending LLM with cross-modality capacity and shed light on the LLM-based ASR community.
hub tools
years
2026 19representative citing papers
Phoneme-based interfaces match or surpass projector-based ones for LLM ASR, especially in low-resource languages, and a BPE-phoneme hybrid offers additional improvements.
A learnable prompt projector added to LLM-based ASR reduces prompt sensitivity, lowers performance variability, and beats the best fixed prompts on four datasets.
On synthetic banking speech alone, GRPO cuts ASR WER 40% relative to SFT (36.71%→22.09%) by improving stopping calibration and attention anchoring to audio.
Learned pooling of speech KV caches from an intermediate LLM layer compresses speech to text-level length while matching or exceeding the uncompressed baseline on ASR and entity recognition, with 1.49–2× decoding speedup.
Proposes an entropy-aware domain-routed MoE Speech-LLM with C-DR, MoP, MoL, and EAR that improves multi-domain child-adult ASR while preserving adult performance.
ELF-S2T applies audio-conditioned flow-matching on continuous text latents from pre-trained ELF to achieve competitive ASR and S2TT results, with analysis showing shared close-distance confusion in latent space.
TRADE augments multimodal Speech LLMs with a transducer branch for streaming ASR, reporting 6.71% WER offline and 8.40% streaming on the Open ASR Leaderboard from one checkpoint.
Common-word acoustic cues and bias-word position prediction in speech LLMs cut rare-word transcription errors by 16.3% versus baselines, including out-of-domain cases.
With matched data and sizes, CTC+LLM shallow fusion beats tight speech-LLM integration on in-domain ASR, while prefix LLMs win average WER on out-of-domain HuggingFace sets.
RespiraMFM reports 9.15% AUROC gain in supervised fine-tuning and 20.98% in zero-shot settings over baselines by aligning respiratory audio with clinical text across seven real-world datasets for five diseases.
The authors introduce LLM-based semantic judgment and an agentic interaction loop that improves semantic fidelity and enables iterative corrections in automatic speech recognition beyond traditional WER.
Mixed batching with only 10% target-domain speech achieves word error rates matching or exceeding conventional full-dataset ASR fine-tuning in LLM-based models.
Feedback-driven alignment with numerical rewards improves MusicLLM emotion regression on arousal and valence over instruction tuning alone while preserving MusicQA performance.
MoE and CIF components are added to an LLM-ASR system and reported to deliver substantial multilingual performance gains over baselines.
SoulX-Transcriber is a unified LLM framework for end-to-end multi-speaker transcription using two-stage training (speaker-aware pre-training then supervised fine-tuning) that reports strong results on AliMeeting, AISHELL-4, and AMI.
A framework using speech-text alignment to generate expressive pseudo-audio prompts for improved text-only domain adaptation in LLM-based ASR.
Lightweight training strategies allow speech-aware LLMs to output accurate word timestamps alongside ASR transcripts while also improving recognition quality across datasets.
Translation-enhanced pre-training of speech encoders improves cross-modal integration and performance in downstream Speech LLM tasks by encouraging language-agnostic representations.
citing papers explorer
-
Speaker Attributed Automatic Speech Recognition Using Speech Aware LLMS
Adapting speech-aware LLMs with speaker cluster identification tags and concatenated multi-speaker data yields superior speaker-attributed ASR performance versus sequential diarization-plus-ASR pipelines.
-
Phonemes vs. Projectors: An Investigation of Speech-Language Interfaces for LLM-based ASR
Phoneme-based interfaces match or surpass projector-based ones for LLM ASR, especially in low-resource languages, and a BPE-phoneme hybrid offers additional improvements.
-
Reducing Prompt Sensitivity in LLM-based Speech Recognition Through Learnable Projection
A learnable prompt projector added to LLM-based ASR reduces prompt sensitivity, lowers performance variability, and beats the best fixed prompts on four datasets.
-
When Synthetic Speech Is All You Have: Better Call GRPO
On synthetic banking speech alone, GRPO cuts ASR WER 40% relative to SFT (36.71%→22.09%) by improving stopping calibration and attention anchoring to audio.
-
Compress the Cache, Not the Speech Embedding: KV Compression for Efficient Speech LLMs
Learned pooling of speech KV caches from an intermediate LLM layer compresses speech to text-level length while matching or exceeding the uncompressed baseline on ASR and entity recognition, with 1.49–2× decoding speedup.
-
Entropy-Aware Domain-Routed Mixture-of-Experts Speech-LLM Framework: A Case Study of Multi-Domain Child-Adult ASR
Proposes an entropy-aware domain-routed MoE Speech-LLM with C-DR, MoP, MoL, and EAR that improves multi-domain child-adult ASR while preserving adult performance.
-
Speech Meets ELF: Audio Conditional Continuous-Target Diffusion for Speech Recognition and Translation
ELF-S2T applies audio-conditioned flow-matching on continuous text latents from pre-trained ELF to achieve competitive ASR and S2TT results, with analysis showing shared close-distance confusion in latent space.
-
TRADE: Transducer-Augmented Decoder for Speech LLM
TRADE augments multimodal Speech LLMs with a transducer branch for streaming ASR, reporting 6.71% WER offline and 8.40% streaming on the Open ASR Leaderboard from one checkpoint.
-
Contextual Biasing for ASR in Speech LLM with Common Word Cues and Bias Word Position Prediction
Common-word acoustic cues and bias-word position prediction in speech LLMs cut rare-word transcription errors by 16.3% versus baselines, including out-of-domain cases.
-
LLMs and Speech: Integration vs. Combination
With matched data and sizes, CTC+LLM shallow fusion beats tight speech-LLM integration on in-domain ASR, while prefix LLMs win average WER on out-of-domain HuggingFace sets.
-
RespiraMFM: A Multimodal Foundation Model with Contrastive Audio-Language Alignment for Respiratory Disease Identification
RespiraMFM reports 9.15% AUROC gain in supervised fine-tuning and 20.98% in zero-shot settings over baselines by aligning respiratory audio with clinical text across seven real-world datasets for five diseases.
-
Interactive ASR: Towards Human-Like Interaction and Semantic Coherence Evaluation for Agentic Speech Recognition
The authors introduce LLM-based semantic judgment and an agentic interaction loop that improves semantic fidelity and enables iterative corrections in automatic speech recognition beyond traditional WER.
-
Closing the Speech-Text Gap with Limited Audio for Effective Domain Adaptation in LLM-Based ASR
Mixed batching with only 10% target-domain speech achieves word error rates matching or exceeding conventional full-dataset ASR fine-tuning in LLM-based models.
-
Aligning MusicLLM with Emotion using Instruction Tuning and Feedback-Driven Alignment
Feedback-driven alignment with numerical rewards improves MusicLLM emotion regression on arousal and valence over instruction tuning alone while preserving MusicQA performance.
-
Enhancing Multilingual LLM-based ASR with Mixture of Experts and Dynamic Downsampling
MoE and CIF components are added to an LLM-ASR system and reported to deliver substantial multilingual performance gains over baselines.
-
SoulX-Transcriber: A Robust End-to-End Framework for Multi-Speaker Speech Transcription
SoulX-Transcriber is a unified LLM framework for end-to-end multi-speaker transcription using two-stage training (speaker-aware pre-training then supervised fine-tuning) that reports strong results on AliMeeting, AISHELL-4, and AMI.
-
Refining Pseudo-Audio Prompts with Speech-Text Alignment for Text-Only Domain Adaptation in LLM-Based ASR
A framework using speech-text alignment to generate expressive pseudo-audio prompts for improved text-only domain adaptation in LLM-based ASR.
-
In-Sync: Adaptation of Speech Aware Large Language Models for ASR with Word Level Timestamp Predictions
Lightweight training strategies allow speech-aware LLMs to output accurate word timestamps alongside ASR transcripts while also improving recognition quality across datasets.
-
Does Translation-Enhanced Speech Encoder Pre-training Affect Speech LLMs?
Translation-enhanced pre-training of speech encoders improves cross-modal integration and performance in downstream Speech LLM tasks by encouraging language-agnostic representations.