pith. sign in

arxiv: 2504.09081 · v2 · pith:BEZ36E7Qnew · submitted 2025-04-12 · 📡 eess.AS · cs.AI· cs.CL

SIFT-50M: A Large-Scale Multilingual Dataset for Speech Instruction Fine-Tuning

Prabhat Pandey , Rupak Vignesh Swaminathan , K V Vijay Girish , Arunasish Sen , Jian Xie , Grant P. Strimel , Andreas Schwarz This is my paper
classification 📡 eess.AS cs.AIcs.CL
keywords speechdatasetllmsfine-tuninginstructionsift-50mspeech-textdesigned
0
0 comments X
read the original abstract

We introduce SIFT (Speech Instruction Fine-Tuning), a 50M-example dataset designed for instruction fine-tuning and pre-training of speech-text large language models (LLMs). SIFT-50M is built from publicly available speech corpora, which collectively contain 14K hours of speech, and leverages LLMs along with off-the-shelf expert models. The dataset spans five languages, encompassing a diverse range of speech understanding as well as controllable speech generation instructions. Using SIFT-50M, we train SIFT-LLM, which outperforms existing speech-text LLMs on instruction-following benchmarks while achieving competitive performance on foundational speech tasks. To support further research, we also introduce EvalSIFT, a benchmark dataset specifically designed to evaluate the instruction-following capabilities of speech-text LLMs.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. MECAT: A Multi-Experts Constructed Benchmark for Fine-Grained Audio Understanding Tasks

    eess.AS 2025-07 unverdicted novelty 7.0

    MECAT is a multi-expert benchmark for audio AI offering fine-grained captions and QA pairs generated via expert models and LLM reasoning, paired with the DATE metric that combines semantic similarity and cross-sample ...

  2. Audio Flamingo Next: Next-Generation Open Audio-Language Models for Speech, Sound, and Music

    cs.SD 2026-04 unverdicted novelty 6.0

    AF-Next is a scaled audio-language model with long-context support and Temporal Audio Chain-of-Thought reasoning that outperforms prior open models on audio understanding and reasoning benchmarks.

  3. Towards Holistic Evaluation of Large Audio-Language Models: A Comprehensive Survey

    eess.AS 2025-05 accept novelty 6.0

    The survey introduces a four-category taxonomy for LALM evaluations and reviews benchmarks across general auditory processing, knowledge reasoning, dialogue, and fairness-safety.

  4. Adaptive Turn-Taking for Real-time Multi-Party Voice Agents

    eess.AS 2026-06 unverdicted novelty 5.0

    ModeratorLM conditions a streaming speech LLM on assigned roles for adaptive turn-taking in multi-party settings, reporting over 40% higher precision and 70% higher recall than non-role baselines on real meetings and ...

  5. Adaptive Turn-Taking for Real-time Multi-Party Voice Agents

    eess.AS 2026-06 unverdicted novelty 4.0

    ModeratorLM conditions a chunk-wise streaming speech LLM on assigned roles (with optional CoT) to raise turn-taking precision over 40% and recall over 70% versus non-role baselines on synthetic RolePlayConv data and r...