Pith. sign in

REVIEW 28 cited by

WavLLM: Towards Robust and Adaptive Speech Large Language Model

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2404.00656 v3 pith:J3AKKZS4 submitted 2024-03-31 cs.CL cs.AIcs.SDeess.AS

Shujie Hu , Long Zhou , Shujie Liu , Sanyuan Chen , Lingwei Meng , Hongkun Hao , Jing Pan , Xunying Liu
show 4 more authors
Jinyu Li Sunit Sivasankaran Linquan Liu Furu Wei
This is my paper · ORCID
classification cs.CLcs.AIcs.SDeess.AS
keywords tasksspeechmodellanguagewavllmcapabilitiescomplexgaokao
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

The recent advancements in large language models (LLMs) have revolutionized the field of natural language processing, progressively broadening their scope to multimodal perception and generation. However, effectively integrating listening capabilities into LLMs poses significant challenges, particularly with respect to generalizing across varied contexts and executing complex auditory tasks. In this work, we introduce WavLLM, a robust and adaptive speech large language model with dual encoders, and a prompt-aware LoRA weight adapter, optimized by a two-stage curriculum learning approach. Leveraging dual encoders, we decouple different types of speech information, utilizing a Whisper encoder to process the semantic content of speech, and a WavLM encoder to capture the unique characteristics of the speaker's identity. Within the curriculum learning framework, WavLLM first builds its foundational capabilities by optimizing on mixed elementary single tasks, followed by advanced multi-task training on more complex tasks such as combinations of the elementary tasks. To enhance the flexibility and adherence to different tasks and instructions, a prompt-aware LoRA weight adapter is introduced in the second advanced multi-task training stage. We validate the proposed model on universal speech benchmarks including tasks such as ASR, ST, SV, ER, and also apply it to specialized datasets like Gaokao English listening comprehension set for SQA, and speech Chain-of-Thought (CoT) evaluation set. Experiments demonstrate that the proposed model achieves state-of-the-art performance across a range of speech tasks on the same model size, exhibiting robust generalization capabilities in executing complex tasks using CoT approach. Furthermore, our model successfully completes Gaokao tasks without specialized training. The codes, models, audio, and Gaokao evaluation set can be accessed at \url{aka.ms/wavllm}.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 28 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Evaluation of LLMs in Speech is Often Flawed: Test Set Contamination in Large Language Models for Speech Recognition

    eess.AS 2025-05 conditional novelty 7.0 of 10

    LibriSpeech and Common Voice evaluation sentences leak into the Pile, and controlled LLM pretraining experiments show that contamination biases output probabilities even when error rates barely change.

  2. Compress the Cache, Not the Speech Embedding: KV Compression for Efficient Speech LLMs

    eess.AS 2026-07 conditional novelty 6.0 of 10

    Learned pooling of speech KV caches from an intermediate LLM layer compresses speech to text-level length while matching or exceeding the uncompressed baseline on ASR and entity recognition, with 1.49–2× decoding speedup.

  3. SPEARBench: A Benchmark for Naturalness Evaluation in Streaming Speech-to-Speech Language Models

    cs.CL 2026-07 conditional novelty 6.0 of 10

    An open-source benchmark for speech-to-speech models shows that current systems produce intelligible audio but diverge from human conversational behavior in latency, dialect consistency, emotional entrainment, and prosody.

  4. Adaptive Perturbation Selection for Contrastive Audio Decoding

    cs.SD 2026-06 unverdicted novelty 6.0 of 10

    A learned per-example router over a 105-perturbation audio library improves contrastive decoding for audio-LLM hallucination, with task-dependent best distortions (e.g., reverse audio for temporal order).

  5. Self-Improvement for Audio Large Language Model using Unlabeled Speech

    cs.SD 2025-07 conditional novelty 6.0 of 10

    SI-SDA uses attention-saliency prompt-reliance as an unsupervised reward to reinforce a model's own beam hypotheses, improving audio LLM WER and BLEU across ASR, S2TT, and SQA without labeled data.

  6. Attacker's Noise Can Manipulate Your Audio-based LLM in the Real World

    cs.CR 2025-07 conditional novelty 6.0 of 10

    Adversarial audio noise, optimized with audio augmentations, can trigger and distort the behavior of audio-based LLMs both digitally and when played through the air.

  7. Zero-Shot Cognitive Impairment Detection from Speech Using AudioLLM

    cs.SD 2025-06 conditional novelty 6.0 of 10

    A zero-shot AudioLLM can classify speech as cognitively normal or impaired with accuracy close to supervised baselines, although prompt selection and small margins temper the claim.

  8. NAVER LABS Europe Submission to the Instruction-following Track

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A separately-trained speech projector and text LoRA can be merged with only 1K steps of multimodal fine-tuning to yield a competitive ASR, ST, and spoken QA system.

  9. Zero-Shot Streaming Text to Speech Synthesis with Transducer and Auto-Regressive Modeling

    cs.LG 2025-05 conditional novelty 6.0 of 10

    SMLLE generates speech frame-by-frame using a Transducer for streaming semantic tokens plus a fully autoregressive mel-spectrogram model, reaching quality close to sentence-level zero-shot TTS.

  10. Speech-IFEval: Evaluating Instruction-Following and Quantifying Catastrophic Forgetting in Speech-Aware Language Models

    eess.AS 2025-05 conditional novelty 6.0 of 10

    Most speech-aware language models follow written output-format instructions far worse than their text-only base LLMs, and Speech-IFEval measures this as catastrophic forgetting.

  11. Vox-Profile: A Speech Foundation Model Benchmark for Characterizing Diverse Speaker and Speech Traits

    cs.SD 2025-05 conditional novelty 6.0 of 10

    A unified multi-trait speech benchmark, Vox-Profile, evaluates foundation models on age, sex, accent, emotion, voice quality, fluency, and expressiveness, and demonstrates downstream uses in ASR analysis and speech ge...

  12. Audio Large Language Models Can Be Descriptive Speech Quality Evaluators

    cs.SD 2025-01 conditional novelty 6.0 of 10

    Audio LLMs fine-tuned with token-level distillation against an LLM teacher can predict speech quality scores and generate natural-language descriptions, including A/B comparisons.

  13. LLM supervised Pre-training for Multimodal Emotion Recognition in Conversations

    eess.AS 2025-01 conditional novelty 6.0 of 10

    A three-stage speech and text model with LLM-generated pseudo-labels on ASR transcripts reports state-of-the-art weighted F1 on MELD and CMU-MOSI.

  14. Advancing Singlish Understanding: Bridging the Gap with Datasets and Multimodal Models

    cs.CL 2025-01 conditional novelty 6.0 of 10

    The authors release MNSC, the largest standardized multitask spoken Singlish corpus, and SingAudioLLM, a multimodal model that sets strong baselines on ASR, spoken QA, dialogue summarization, and paralinguistic QA.

  15. A Unified Denoising and Adaptation Framework for Self-Supervised Bengali Dialectal ASR

    cs.SD 2025-08 conditional novelty 5.0 of 10

    A two-stage, noise-augmented fine-tuning of WavLM achieves state-of-the-art WER/CER on Bengali dialectal ASR under clean and noisy conditions.

  16. SpeechLLM: Unified Speech and Language Model for Enhanced Multi-Task Understanding in Low Resource Settings

    cs.CL 2025-08 conditional novelty 5.0 of 10

    A parameter-efficient adapter bridging Whisper and TinyLlama reports relative improvements in speech recognition, named entity recognition, and sentiment analysis on low-resource benchmarks.

  17. OmniCharacter: Towards Immersive Role-Playing Agents with Seamless Speech-Language Personality Interaction

    cs.CL 2025-05 conditional novelty 5.0 of 10

    OmniCharacter is a speech-language role-playing agent that generates character-specific voice responses with low latency, trained on a new 10K-dialogue, 135K-audio dataset of 20 game characters.

  18. ALAS: An Automatic Latent Alignment Score for Audio Language Models

    cs.CL 2025-05 conditional novelty 5.0 of 10

    ALAS is a reference-based score for audio-text alignment in speech LLMs, computed from frozen hidden states and a Whisper-derived alignment path, with no training or fitted classifier.

  19. SpokenNativQA: Multilingual Everyday Spoken Queries for LLMs

    cs.CL 2025-05 conditional novelty 5.0 of 10

    SpokenNativQA is a human-recorded Arabic and English spoken question-answering benchmark built from MultiNativQA text pairs, with ASR and LLM baselines.

  20. LiSTEN: Learning Soft Token Embeddings for Neural Audio LLMs

    cs.AI 2025-05 conditional novelty 5.0 of 10

    LiSTEN shows that dynamically selecting a few learnable prompt tokens from a shared pool can replace LoRA fine-tuning for audio-language models, matching or beating it with less training data.

  21. A Preliminary Exploration with GPT-4o Voice Mode

    cs.CL 2025-02 conditional novelty 5.0 of 10

    GPT-4o voice mode is evaluated on 180 Dynamic-SUPERB tasks plus MMAU and CMM, showing strong audio understanding and low hallucination, but unstable refusal behavior and weak duration and instrument skills.

  22. Prepending or Cross-Attention for Speech-to-Text? An Empirical Comparison

    cs.CL 2025-01 conditional novelty 5.0 of 10

    Across controlled ASR and speech translation experiments, dense feature prepending does not outperform cross-attention in quality and is slightly slower and more memory hungry.

  23. BEST-STD: Bidirectional Mamba-Enhanced Speech Tokenization for Spoken Term Detection

    eess.AS 2024-11 conditional novelty 5.0 of 10

    BEST-STD trains a bidirectional Mamba encoder with contrastive learning and vector quantization to make speaker-agnostic speech tokens for fast spoken term detection.

  24. Toward Generalizable Cognitive Impairment Detection with Speech-Based Multimodal Large Language Models

    eess.SP 2026-07 reject novelty 4.0 of 10

    A multimodal LLM pipeline (Qwen audio + Qwen text embeddings, concatenated and classified) reaches 92.4% accuracy on a combined ADReSS20/ADReSSo21 test set, but the evaluation does not justify state-of-the-art or cros...

  25. NAVER LABS System Re-implementation for the IWSLT 2026 Instruction-Following Task

    cs.CL 2026-07 conditional novelty 4.0 of 10

    Open re-implementation of the NAVER LABS three-stage speech-LLM pipeline for IWSLT 2026 yields COMET 0.781 EN–ZH ST and BERTScore-F1 0.346 English SQA, with 100k synthetic data released.

  26. Overview of the Amphion Toolkit (v0.2)

    cs.SD 2025-01 conditional novelty 4.0 of 10

    Amphion v0.2 is an open-source toolkit for audio, music, and speech generation, adding a 101K-hour multilingual dataset, processing pipelines, and pretrained models.

  27. Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Survey

    cs.CL 2024-12 conditional novelty 4.0 of 10

    A comprehensive survey that frames multimodal understanding and generation as next token prediction and proposes a five-part taxonomy.

  28. WavChat: A Survey of Spoken Dialogue Models

    eess.AS 2024-11 conditional novelty 4.0 of 10

    WavChat categorizes spoken dialogue models into cascaded and end-to-end paradigms and surveys speech representations, training strategies, streaming, duplex interaction, datasets, and evaluation benchmarks.

Pith tools