Pith. sign in

REVIEW 20 cited by

Spirit LM: Interleaved Spoken and Written Language Model

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2402.05755 v2 pith:UN3NDKLH submitted 2024-02-08 cs.CL cs.SDeess.AS

Tu Anh Nguyen , Benjamin Muller , Bokai Yu , Marta R. Costa-jussa , Maha Elbayad , Sravya Popuri , Christophe Ropers , Paul-Ambroise Duquenne
show 8 more authors
Robin Algayres Ruslan Mavlyutov Itai Gat Mary Williamson Gabriel Synnaeve Juan Pino Benoit Sagot Emmanuel Dupoux
This is my paper · ORCID
classification cs.CLcs.SDeess.AS
keywords speechmodeltextspiritunitslanguagemodelsabilities
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

We introduce Spirit LM, a foundation multimodal language model that freely mixes text and speech. Our model is based on a 7B pretrained text language model that we extend to the speech modality by continuously training it on text and speech units. Speech and text sequences are concatenated as a single stream of tokens, and trained with a word-level interleaving method using a small automatically-curated speech-text parallel corpus. Spirit LM comes in two versions: a Base version that uses speech phonetic units (HuBERT) and an Expressive version that models expressivity using pitch and style units in addition to the phonetic units. For both versions, the text is encoded with subword BPE tokens. The resulting model displays both the semantic abilities of text models and the expressive abilities of speech models. Additionally, we demonstrate that Spirit LM can learn new tasks in a few-shot fashion across modalities (i.e. ASR, TTS, Speech Classification). We make available model weights and inference code.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 20 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Scaling Speech-Text Pre-training with Synthetic Interleaved Data

    cs.CL 2024-11 conditional novelty 7.0 of 10

    Synthetic interleaved speech-text data generated from text corpora lets a 9B speech LM reach SOTA spoken QA (31% vs 13% previous average SOTA).

  2. On the Fallacy of Global Token Perplexity in Spoken Language Model Evaluation

    cs.CL 2026-01 conditional novelty 6.0 of 10

    Global token perplexity mis-ranks spoken language models; localized/normalized likelihood scores and an embedding judge track human MOS better and make the best model look much closer to human.

  3. Autoregressive Speech Enhancement via Acoustic Tokens

    cs.SD 2025-07 conditional novelty 6.0 of 10

    Acoustic tokens outperform semantic tokens on speaker identity in speech enhancement, an autoregressive transducer helps in some settings, but discrete representations still lag continuous ones.

  4. OpusLM: A Family of Open Unified Speech Language Models

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A new open family of speech-language models trained on public data matches or beats prior systems across ASR, TTS, and text benchmarks.

  5. NTPP: Generative Speech Language Modeling for Dual-Channel Spoken Dialogue via Next-Token-Pair Prediction

    cs.CL 2025-06 conditional novelty 6.0 of 10

    NTPP models dual-channel spoken dialogue by predicting both speakers' next speech tokens as a pair, achieving speaker-independent full-duplex generation in a decoder-only transformer.

  6. Exploring the Effect of Segmentation and Vocabulary Size on Speech Tokenization for Speech Language Models

    cs.CL 2025-05 conditional novelty 6.0 of 10

    Using 80 ms speech segments and 16,384 sound tokens improves zero-shot spoken language understanding and cuts training cost by up to 70%.

  7. VoxEval: Benchmarking the Knowledge Understanding Capabilities of End-to-End Spoken Language Models

    cs.CL 2025-01 conditional novelty 6.0 of 10

    A speech-only QA benchmark built from MMLU shows that current end-to-end spoken language models perform near or below random guessing and are brittle to audio changes.

  8. Advancing Singlish Understanding: Bridging the Gap with Datasets and Multimodal Models

    cs.CL 2025-01 conditional novelty 6.0 of 10

    The authors release MNSC, the largest standardized multitask spoken Singlish corpus, and SingAudioLLM, a multimodal model that sets strong baselines on ASR, spoken QA, dialogue summarization, and paralinguistic QA.

  9. Interleaved Speech-Text Language Models for Simple Streaming Text-to-Speech Synthesis

    eess.AS 2024-12 conditional novelty 6.0 of 10

    Training a decoder-only LM on fixed-ratio interleaved text and speech tokens yields a simple zero-shot streaming TTS, with a 1:3 text-to-speech chunk ratio keeping WER within about 8 percent relative of non-streaming.

  10. Contrastive Learning for Task-Independent SpeechLLM-Pretraining

    cs.CL 2024-12 conditional novelty 6.0 of 10

    Contrastive pre-training that aligns speech and text across all model layers beats ASR-based pre-training and, with 10% of task data, matches or exceeds specialized models on translation and question answering.

  11. 2M-BELEBELE: Highly Multilingual Speech and American Sign Language Comprehension Dataset

    cs.CL 2024-12 conditional novelty 6.0 of 10

    2M-BELEBELE is a new multilingual speech and ASL comprehension benchmark built from BELEBELE and FLEURS, with human recordings for 74 spoken languages and ASL video with glosses.

  12. TouchTTS: An Embarrassingly Simple TTS Framework that Everyone Can Touch

    cs.SD 2024-12 conditional novelty 6.0 of 10

    TouchTTS reports a 51.6% data retention rate using a noise-robust tokenizer and two-ASR cross-validation, and a Qwen-backbone flow model that unifies streaming and non-streaming synthesis while matching CosyVoice on PER.

  13. An Empirical Analysis of Discrete Unit Representations in Speech Language Modeling Pre-training

    cs.CL 2025-09 conditional novelty 5.0 of 10

    Smaller discrete vocabularies (k = 125 to 1,000), WavLM units, and larger models give the lowest negative log-likelihood in speech language model pre-training.

  14. Seeing is Believing: Emotion-Aware Audio-Visual Language Modeling for Expressive Speech Generation

    cs.CL 2025-08 conditional novelty 5.0 of 10

    An audio-visual language model that adds full-face visual features to a pre-trained expressive speech model improves emotion recognition and expressive speech generation by a few F1 points over speech-only on syntheti...

  15. Enhancing Generalization of Speech Large Language Models with Multi-Task Behavior Imitation and Speech-Text Interleaving

    eess.AS 2025-05 conditional novelty 5.0 of 10

    Multi-task behavior imitation with speech-text interleaving improves speech LLM generalization on prompts and zero-shot tasks using only paired speech and transcripts.

  16. Voila: Voice-Language Foundation Models for Real-Time Autonomous Interaction and Voice Role-Play

    cs.AI 2025-05 reject novelty 5.0 of 10

    Voila presents a unified voice-language model family with a hierarchical multi-scale Transformer, claiming 195 ms latency and a million customizable voices, with new benchmark and ASR/TTS evaluations.

  17. LLaMA-Omni2: LLM-based Real-time Spoken Chatbot with Autoregressive Streaming Speech Synthesis

    cs.CL 2025-05 conditional novelty 5.0 of 10

    A series of Qwen2.5-based speech chatbots (0.5B to 14B) that stream speech output with a CosyVoice2-style autoregressive decoder and outperform prior SpeechLMs using only 200K synthetic multi-turn dialogues.

  18. SLAM-Omni: Timbre-Controllable Voice Interaction System with Single-Stage Training

    eess.AS 2024-12 conditional novelty 5.0 of 10

    A 0.5B spoken dialogue model trained end-to-end in one stage, with grouped semantic tokens for faster generation and text-only history for multi-turn dialogue.

  19. SALMONN-omni: A Codec-free LLM for Full-duplex Speech Understanding and Generation

    eess.AS 2024-11 reject novelty 5.0 of 10

    SALMONN-omni proposes a codec-free, full-duplex speech LLM with state-transition 'thinking' tokens, but provides only qualitative case studies as evidence.

  20. WavChat: A Survey of Spoken Dialogue Models

    eess.AS 2024-11 conditional novelty 4.0 of 10

    WavChat categorizes spoken dialogue models into cascaded and end-to-end paradigms and surveys speech representations, training strategies, streaming, duplex interaction, datasets, and evaluation benchmarks.

Pith tools