WavChat: A Survey of Spoken Dialogue Models

Hanting Wang; Jialong Zuo; Jian Li; Jingyu Lu; Jingzhen He; Jin Xu; Long Zhou; Minghui Fang; Qian Yang; Shengpeng Ji

REVIEW 3 major objections 6 minor 26 cited by

WavChat: A Survey of Spoken Dialogue Models

T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This survey organizes spoken dialogue systems into cascaded and end-to-end paradigms and maps the technologies behind them.

desk verdict Useful survey with a taxonomy that doesn't always apply; worth reviewing after fixing the categorization and selection transparency. read the letter →

arxiv 2411.13577 v2 pith:AZVUUSCO submitted 2024-11-15 eess.AS cs.CLcs.LGcs.MMcs.SD

Shengpeng Ji , Yifu Chen , Minghui Fang , Jialong Zuo , Jingyu Lu , Hanting Wang , Ziyue Jiang , Long Zhou

show 11 more authors

Shujie Liu Xize Cheng Xiaoda Yang Zehan Wang Qian Yang Jian Li Yidi Jiang Jingzhen He Yunfei Chu Jin Xu Zhou Zhao

This is my paper · ORCID

classification eess.AScs.CLcs.LGcs.MMcs.SD

keywords spokendialoguemodelscascadedspeechsystemsend-to-endrepresentationsfull-duplexinteractionstreamingtrainingparadigmsevaluationbenchmarks

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This survey sets out to give the field of spoken dialogue models its first systematic map. It compiles the systems released in recent years in chronological order and splits them into two paradigms: cascaded models, which route speech through text using ASR, an LLM, and TTS, and end-to-end models, whose core language model directly understands and generates speech representations. Around that split, it organizes the underlying technology into four areas—speech representation, training paradigm, streaming/duplex/interaction, and data/evaluation—and closes each with open problems. The value of the map, if it holds, is that researchers and engineers can locate any system on a common grid, compare design choices, and see where the gaps are. The paper does not claim a new model or a measured result; its contribution is the organization and vocabulary.

What carries the argument

The load-bearing device is the cascaded/end-to-end dichotomy, defined by whether the core LLM can directly understand and generate speech representations rather than relying on text as an intermediate. The survey's secondary machinery is a four-part decomposition of the field: speech representations split into semantic tokens (HuBERT, Whisper, S3) and acoustic tokens (EnCodec, SpeechTokenizer, Mimi); training paradigms arranged as five architecture types plus a four-stage pipeline (text LLM pretraining, modality alignment, supervised fine-tuning, preference optimization); streaming and interaction techniques such as causal convolution, causal attention, queue management, and simplex/half-/full-duplex modes; and a catalogue of datasets, evaluation metrics, and benchmarks. This grid carries the argument by giving every surveyed system a location and making the field's open problems visible.

What would settle it

A reader could run a systematic search for spoken-dialogue systems announced in the same period and count how many influential ones are missing from the paper's timeline, especially releases without published papers that the text itself names; if the omissions change the narrative of how the field progressed, the claim to a complete chronological compilation is weakened.

Watch

Extended reading notes

Core claim

The paper's central claim is that the rapid recent growth in spoken dialogue systems can be understood through one organizing question: does the core language model directly understand and generate speech representations? Systems that answer yes are classified end-to-end; systems that only generate text and convert it to speech are classified cascaded. Using this criterion, WavChat assembles the systems chronologically and then reviews four interlocking technology areas (speech representations, training paradigms, streaming/duplex/interaction design, and training/evaluation resources), describing limitations and future directions in each. The survey presents itself as the first to organize the field this way and as a reference point for both academic research and industrial applications.

Load-bearing premise

The survey assumes that the systems it chose to include, selected without a documented search or inclusion protocol, are representative enough of the whole field to support a comprehensive chronological map and technology review.

Editorial extensions

If this is right

New systems can be classified quickly by whether the core LLM directly consumes or produces speech tokens, making comparisons across papers more systematic.
The four-part grid functions as a design checklist: representation choice, alignment and training strategy, streaming and duplex handling, and evaluation plan.
The survey identifies concrete open gaps: no unified framework for all interaction forms, scarce RL and preference optimization in spoken dialogue, few models beyond English and Chinese, and underdeveloped audio/music generation and security evaluation.
The chronological timeline frames the field's trajectory as moving from text-intermediary pipelines (AudioGPT) through speech-token LLMs (SpeechGPT) to full-duplex streaming systems (Moshi, Mini-Omni2).
Because most systems are built on pretrained text LLMs with speech-text alignment, the survey implies that preserving text intelligence while adding speech is the central training problem.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The binary cascaded/end-to-end split may become less crisp as cascaded systems gain trainable streaming TTS and end-to-end systems keep implicit text guidance; a graded degree-of-text-mediation axis might be a more faithful map.
The exclusion from the timeline of systems without published papers (e.g., Westlake-Omni, MooER-Omni, Hertz-dev, SpeechGPT2, Fish-Agent) means the map can lag the deployed frontier; tracking paper-less releases as they appear would test the timeline's completeness.
The survey's representation discussion suggests a testable prediction: codecs that distill semantic information into the first quantizer layer (as SpeechTokenizer and Mimi do) will become the default speech tokenizer for end-to-end dialogue, because they resolve the semantic-versus-acoustic tradeoff before the LLM sees the tokens.
The absence of RL and preference optimization in spoken dialogue, contrasted with its role in TTS, points to an extension: preference data defined on interaction outcomes (interrupt handling, backchannels, latency) rather than speech quality alone.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Dear colleague,

The first thing you should know about WavChat is that it is genuinely useful: it is the first survey I know that puts the recent explosion of spoken dialogue models into one chronological map, and the four-part organization (representations, training, streaming/duplex, data and evaluation) is a sensible frame. The dataset and benchmark tables alone are worth the download. But the central taxonomy, the cascaded/end-to-end split, is not applied as cleanly as the paper's own definition promises. The definition in Section 2.3 says end-to-end means the core LLM directly understands and generates speech representations. Then Freeze-Omni is called end-to-end even though the paper says its core idea is to move the functions to the ASR encoder and TTS decoder rather than the LLM, and Llama-Omni is listed as end-to-end while its speech decoder generates units from LLM hidden states outside the LLM's output layer. SpeechGPT is also end-to-end by the paper's classification, even though it generates text before speech tokens. So 'end-to-end' is doing different work in different places. This matters because the whole value of a survey like this is that a reader can use the map without having to rederive it. As written, I couldn't reproduce the figure's assignments from the stated rule. That is a real weakness, and I agree with the stress-test that it is the biggest single risk to the paper's usefulness.

There are also smaller issues: no documented selection protocol, Figure 1 omits several named systems (Westlake-Omni, MooER-Omni, etc.), and there are citation inconsistencies (Freeze-Omni is [213] in one place and [214] in another). The paper is marked 'Working in Progress', which explains some of this, but a survey that claims comprehensiveness needs to say how it chose its systems.

What the paper does well is substantial. The discussions of semantic vs acoustic representations, the multi-stage training breakdown, and the streaming/duplex interaction analysis are accurate and well-sourced. The chronological timeline up to late 2024 is useful. I did not find evidence of misrepresentation of the underlying systems, aside from the classification issue above.

Who is this for? Someone entering the spoken dialogue area, or a researcher wanting a quick reference for what systems exist and how they are built. It won't change the direction of the field, but it can save a newcomer weeks of digging.

My recommendation: send it to peer review. The taxonomy needs a serious revision (perhaps a third 'hybrid' category or an explicit separation of text-guided vs text-free end-to-end), the selection criteria need to be documented, and the citations need a cleanup. But the underlying work is solid and the field needs this kind of map. If the authors fix the categorization, I would cite it without hesitation.

That's my read.

Referee Report

3 major / 6 minor

Summary. The manuscript surveys spoken dialogue models (SDMs) as of late 2024. It proposes a binary taxonomy that separates cascaded from end-to-end systems based on whether the core language model can directly understand and generate speech representations, presents a chronological timeline of systems in Figure 1, and then reviews speech representations, training paradigms, streaming/duplex/interaction mechanisms, and training/evaluation resources. It does not propose a new model or report new experimental results; its contribution is the organization, comparison, and discussion of existing systems.

Significance. If its taxonomy were applied consistently and its coverage criteria documented, this would be a useful reference for a rapidly growing area. The paper assembles a broad catalog of systems, organizes architectural choices around representations, alignment, training stages, duplex/streaming behavior, and benchmarks, and ends each section with a discussion of open problems. The explicit separation of semantic and acoustic representations and the summary of existing evaluation resources are genuinely useful. No machine-checked proofs or reproducible code are involved because the contribution is a survey; the associated GitHub repository is a helpful pointer but is not itself a reproducibility artifact. The main threat to the paper's value is the inconsistent application of its central categorization, which currently prevents a reader from using the map as the authors intend.

major comments (3)

[§1, §2.3, Figure 2] The paper's central definition of end-to-end is stated as 'whether the core language model can directly understand and generate speech representations' (§1) and repeated in §2.3, yet several systems placed in the end-to-end column do not satisfy this criterion. §2.3 describes Freeze-Omni as transferring 'the functionalities of spoken dialogue models to the encoder (ASR) and decoder (TTS), rather than assigning these tasks to the large language model,' and describes Llama-Omni and IntrinsicVoice as feeding 'the hidden states of the LLM' into a separate decoder. Hidden states are not speech representations, and a TTS decoder is not the LLM generating speech. The same ambiguity appears for SpeechGPT, which generates text before speech (§2.3) and is later described in §3.3.4 as using intermediate text as a semantic guide. The manuscript therefore uses 'end-to-end' to mean at least three different properties: no external ASR/TTS modules, no text intermediacy, and LLM-emitting speech tokens. Please either narrow the definition, introduce a third (hybrid/hidden-state) category, or explicitly classify these systems along all three axes; otherwise the central map cannot be reproduced by a reader.
[§1, Figure 1 caption] The abstract claims that the paper is the first to compile existing spoken dialogue systems in chronological order, but the paper never states a search or inclusion/exclusion protocol. Figure 1's caption says systems without published papers (Westlake-Omni, MooER-Omni, Hertz-dev, SpeechGPT2, Fish-Agent) are excluded even though some are named in the text and are part of the field; the timeline is therefore not a complete map of 'existing' systems. To make the survey's coverage claim falsifiable, please add a scope-and-methods paragraph stating the date of the literature search, the sources searched, the criteria for including a system (e.g., public demo, technical report, checkpoint availability), and how the chronological order is determined (arXiv submission date vs release date). As written, the selection appears to be subjective, and a reader cannot tell whether the omissions are material.
[§2.3, §4.1, §5.1.1] The paper's own subcategorizations conflict with its headline taxonomy. §4.1 places Llama-Omni under 'Parallel Generation of Text and Speech' rather than 'Speech-to-Speech Generation,' while §2.3 places it in the end-to-end group; §5.1.1 states that several systems named in the end-to-end group 'cannot be considered fully streaming because they are not designed to be streaming on the input side.' These statements are not errors in themselves, but they show that the binary 'cascaded vs end-to-end' distinction is orthogonal to at least two other distinctions (text-guided vs text-free; streaming input vs streaming output) that the paper does not separate. Please add a table or figure that crosses the taxonomy with these additional axes so that systems like Llama-Omni, Mini-Omni, and IntrinsicVoice are classified unambiguously.

minor comments (6)

[§2.3, §3.1, §3.2, §5.2.2] Citation numbers for Freeze-Omni are inconsistent: it is cited as [213] in §2.3 and as [214] in §3.1, §3.2, and §5.2.2. Please unify the reference numbers and verify that the bibliography entries correspond.
[Throughout] Several typos should be corrected: 'noly' should be 'only' (§2.3), 'interactation' should be 'interaction' (§5), 'moded' should be 'model' (§5.2.1), and 'futher' should be 'further' (§3.2).
[Figure 3, §4.1, §4.3] System names are not standardized: 'SALMONN' appears as 'SALMMON' (§4.1) and 'SLAMONN' (Figure 3), and 'Llama-Omni' appears as 'LLama-Omni' and 'Lama-Omni' (§4.3). Please standardize names to the official spellings.
[Figure 1] The caption says that publicly available checkpoints are marked in yellow, but the figure does not contain a legend, and the text does not define the marking scheme. Please add a legend and clarify that the timeline includes only systems with published papers, so the reader is not misled about coverage.
[Table 2] The row for Wenetspeech is labeled 'TTS' although WenetSpeech is widely used as an ASR corpus; please correct the task label and re-check all task assignments in the table (e.g., VCTK, Common Voice, and Gigaspeech) for accuracy.
[Title page] The manuscript is marked 'Working in Progress.' This label is not appropriate for a journal submission; please remove it and bring the manuscript to a finished state, including the missing search protocol described in the major comments, before resubmission.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: WavChat is a survey whose taxonomy is a classification rule applied to external published systems, not a derivation or prediction that reduces to its inputs.

full rationale

WavChat is a survey, not a derivation or prediction. Its central claim is that it has compiled spoken dialogue systems chronologically and categorized them as cascaded versus end-to-end by applying the stated criterion: whether the core language model can directly understand and generate speech representations (Section 1, Figure 2, Section 2.3). The taxonomy is a classification rule applied to independently published systems; no result is derived from a fitted parameter, no quantity is predicted from data, and no uniqueness theorem or ansatz is imported via self-citation. The paper's own limitations, such as Figure 1 excluding systems without published papers and Section 5.1.1 noting that IntrinsicVoice is not fully streaming, are coverage and interpretation judgments, not circular reasoning. Although members of the author group contributed to some described systems (e.g., Mini-Omni and Mini-Omni2), the survey's organizational claims do not rest on the correctness of those descriptions; the cited systems are externally published artifacts with their own papers. Potential concerns about taxonomy consistency, such as classification of Freeze-Omni and Llama-Omni under the paper's stated end-to-end definition, are correctness risks about internal consistency, not circularity, because the classification does not define the systems into existence and is not used to derive a predicted outcome. No circular step can be exhibited with a quote and a specific reduction, so the score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No new entities or fitted parameters are introduced because the paper is a survey. The ledger lists the assumptions that make the survey's categories and coverage claims meaningful.

assumptions (3)

domain assumption Spoken dialogue models can be partitioned into cascaded and end-to-end by whether the core language model directly understands and generates speech representations.
Invoked throughout Section 2 and Figure 2; systems that output text only are labeled cascaded even when they accept speech input, and systems that generate text as an inner stream, such as Moshi, are labeled end-to-end. The binary is a modeling choice, not a derived fact.
ad hoc to paper The set of surveyed systems is representative of the field despite undocumented inclusion criteria.
Figure 1's caption excludes systems without published papers even though some are named in the text, and no search or selection protocol is given, yet the paper claims comprehensiveness.
domain assumption The nine usage scenarios defined in Section 2.1 cover the relevant functions of spoken dialogue models.
The scenarios are derived from demos and interfaces of a few representative models, including GPT-4o, Moshi, Qwen2-Audio, and VITA, and are then used as the evaluation framework in Section 6.

how reviews work

0 comments

Cite this review

Pith. "Pith review of WavChat: A Survey of Spoken Dialogue Models." pith.science (2026). https://pith.science/paper/AZVUUSCO

@misc{pith2026241113577,
  author       = {Pith},
  title        = {Pith review of: WavChat: A Survey of Spoken Dialogue Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AZVUUSCO}},
  note         = {Machine review of arXiv:2411.13577}
}

read the original abstract

Recent advancements in spoken dialogue models, exemplified by systems like GPT-4o, have captured significant attention in the speech domain. Compared to traditional three-tier cascaded spoken dialogue models that comprise speech recognition (ASR), large language models (LLMs), and text-to-speech (TTS), modern spoken dialogue models exhibit greater intelligence. These advanced spoken dialogue models not only comprehend audio, music, and other speech-related features, but also capture stylistic and timbral characteristics in speech. Moreover, they generate high-quality, multi-turn speech responses with low latency, enabling real-time interaction through simultaneous listening and speaking capability. Despite the progress in spoken dialogue systems, there is a lack of comprehensive surveys that systematically organize and analyze these systems and the underlying technologies. To address this, we have first compiled existing spoken dialogue systems in the chronological order and categorized them into the cascaded and end-to-end paradigms. We then provide an in-depth overview of the core technologies in spoken dialogue models, covering aspects such as speech representation, training paradigm, streaming, duplex, and interaction capabilities. Each section discusses the limitations of these technologies and outlines considerations for future research. Additionally, we present a thorough review of relevant datasets, evaluation metrics, and benchmarks from the perspectives of training and evaluating spoken dialogue systems. We hope this survey will contribute to advancing both academic research and industrial applications in the field of spoken dialogue systems. The related material is available at https://github.com/jishengpeng/WavChat.

Figures

Figures reproduced from arXiv: 2411.13577 by the authors.

**Figure 2.** A general overview of current spoken dialogue systems. We categorize these systems into [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** A general overview about the structure of WavChat [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: An overall demonstration of the functions of the spoken dialogue systems. We describe [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 5.** Figure 5: Categorization Diagram of Spoken Dialogue Model Architectural Paradigms. [PITH_FULL_IMAGE:figures/full_fig_p019_5.png]

**Figure 6.** Figure 6: Diagram of Multi-stage Training Steps. 4.2.1 Text LLM Pre-Training The goal is to develop a text-intelligent LLM model capable of handling complex contexts and possessing knowledge reasoning abilities, thus preparing it for integration with speech-intelligent LLMs. Mos…

**Figure 7.** Figure 7: Alignment Post-training Methods. speech generation handled by a separate decoder module. SpeechGPT [243], Moshi [44], and VITA [61] utilize unlabeled speech corpora to train models in a next-token prediction task. In the first phase, VITA focuses on training the audio …

**Figure 8.** Figure 8: The illustration of Simplex, Half-Duplex, and Full-Duplex. [PITH_FULL_IMAGE:figures/full_fig_p028_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 26 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Dual Information Speech Language Models for Emotional Conversations
cs.CL 2025-08 conditional novelty 7.0 of 10

A dual-adapter design with equivalence replacement regularization lets frozen LLMs perceive both paralinguistic and linguistic information from speech for emotional conversation.
VoxZip: Semantic-Anchored Temporal KV Cache Compression for Long-Context Audio Inference
cs.AI 2026-08 conditional novelty 6.0 of 10

A train-free, ASR-anchored two-stage KV cache compression method preserves over 90% of uncompressed performance on long-context audio inference at 20x compression.
Hierarchical Acoustic-Semantic Modeling: Modality Separation and Semantic Coherence for Full-Duplex SLMs
cs.CL 2026-07 conditional novelty 6.0 of 10

Lychee-FD resolves modality interference in full-duplex spoken language models by separating acoustic and semantic parameters in deep layers and adding a dense semantic alignment channel, achieving state-of-the-art pe...
TurnNat: Automatic Evaluation of Turn-Taking Naturalness in Dyadic Spoken Dialogue
cs.CL 2026-07 unverdicted novelty 6.0 of 10

TurnNat introduces a likelihood-based automatic evaluation method for turn-taking naturalness in dyadic spoken dialogues using a causal prediction model and a human-validated perturbation benchmark.
A Synonymous Variational Perspective on the Rate-Distortion-Perception Tradeoff
cs.IT 2026-04 unverdicted novelty 6.0 of 10

Synset-based reconstruction and synonymous variational inference are claimed to derive the distributional divergence in RDP and unify it with classical rate-distortion theory.
Towards Accurate Phonetic Error Detection Through Phoneme Similarity Modeling
eess.AS 2025-07 conditional novelty 6.0 of 10

Soft phoneme-similarity labels in multi-task training improve verbatim phoneme recognition and yield two new pronunciation error metrics.
OpusLM: A Family of Open Unified Speech Language Models
cs.CL 2025-06 conditional novelty 6.0 of 10

A new open family of speech-language models trained on public data matches or beats prior systems across ASR, TTS, and text benchmarks.
Streaming Endpointer for Spoken Dialogue using Neural Audio Codecs and Label-Delayed Training
cs.SD 2025-06 conditional novelty 6.0 of 10

Using Mimi neural codec features with label-delayed training reduces endpoint cutoff errors by 42.7% (single-stream) and 37.5% (two-stream) at 160 ms median latency.
BR-ASR: Efficient and Scalable Bias Retrieval Framework for Contextual Biasing ASR in Speech LLM
cs.SD 2025-05 conditional novelty 6.0 of 10

A speech-and-bias contrastive retrieval framework with homophone-aware curriculum learning scales contextual ASR biasing to 200,000 entries while improving B-WER on LibriSpeech.
PersonaTAB: Predicting Personality Traits using Textual, Acoustic, and Behavioral Cues in Fully-Duplex Speech Dialogs
cs.SD 2025-05 conditional novelty 6.0 of 10

An LLM prompted with transcripts, laughter, turn-taking, backchannel, and emotion statistics predicts Big Five personality traits from two-person phone calls, agreeing with human ratings more closely than three text-o...
OmniChat: Enhancing Spoken Dialogue Systems with Scalable Synthetic Data for Diverse Scenarios
cs.CL 2025-01 conditional novelty 6.0 of 10

ShareChatX and OmniChat show that large-scale synthetic spoken dialogue data improves multi-turn response quality and emotion prediction, setting a new state of the art on DailyTalk.
Advancing Singlish Understanding: Bridging the Gap with Datasets and Multimodal Models
cs.CL 2025-01 conditional novelty 6.0 of 10

The authors release MNSC, the largest standardized multitask spoken Singlish corpus, and SingAudioLLM, a multimodal model that sets strong baselines on ASR, spoken QA, dialogue summarization, and paralinguistic QA.
X$^3$-OPD: Distilling Reasoning into Large Audio-Language Models via On-Policy Alignment
cs.LG 2026-07 conditional novelty 5.0 of 10

X3-OPD improves audio-grounded reasoning by training the audio student on its own rollouts with token-level teacher feedback, using a three-tier paired text-audio corpus.
SMART: Shot-Aware Multimodal Video Moment Retrieval with Audio-Enhanced MLLM
cs.CV 2025-11 conditional novelty 5.0 of 10

SMART, an audio-enhanced MLLM with shot-aware token compression, reports new state-of-the-art moment retrieval accuracy on Charades-STA and QVHighlights.
ChipChat: Low-Latency Cascaded Conversational Agent in MLX
eess.AS 2025-08 conditional novelty 5.0 of 10

ChipChat is a fully on-device cascaded voice agent that reports about 920 ms total latency using streaming ASR, a state-action LLM, streaming TTS, and a vocoder.
StreamFlow: Streaming Flow Matching with Block-wise Guided Attention Mask for Speech Token Decoding
cs.SD 2025-06 conditional novelty 5.0 of 10

StreamFlow achieves streaming speech token decoding with 180 ms first-packet latency by using hierarchical block-wise attention masks in a DiT flow matching model.
Step-Audio-AQAA: a Fully End-to-End Expressive Large Audio Language Model
cs.SD 2025-06 conditional novelty 5.0 of 10

Step-Audio-AQAA, a 130B end-to-end audio language model using dual-codebook tokens, text-audio interleaving, masked DPO and weight merging, is claimed to outperform Kimi-Audio and Qwen-Omni on the authors' StepEval-Au...
Speechless: Speech Instruction Training Without Speech for Low Resource Languages
eess.AS 2025-05 conditional novelty 5.0 of 10

Fine-tuning an LLM on text instructions converted to Whisper semantic tokens enables it to understand spoken instructions at inference, bypassing TTS and speech instruction data.
LLaMA-Omni2: LLM-based Real-time Spoken Chatbot with Autoregressive Streaming Speech Synthesis
cs.CL 2025-05 conditional novelty 5.0 of 10

A series of Qwen2.5-based speech chatbots (0.5B to 14B) that stream speech output with a CosyVoice2-style autoregressive decoder and outperform prior SpeechLMs using only 200K synthetic multi-turn dialogues.
LUCY: Linguistic Understanding and Control Yielding Early Stage of Her
cs.CL 2025-01 conditional novelty 5.0 of 10

LUCY uses curated synthetic data to train a Mini-Omni-style speech model that responds to both what you say and how you say it, makes natural short replies, and calls external tools.
Audio-Language Models for Audio-Centric Tasks: A Systematic Survey
cs.SD 2025-01 conditional novelty 5.0 of 10

A systematic survey that categorizes audio-language models by architecture, training objective, and application, covering speech, music, and general audio.
MinMo: A Multimodal Large Language Model for Seamless Voice Interaction
cs.CL 2025-01 conditional novelty 5.0 of 10

A single aligned multimodal model, MinMo, achieves strong or state-of-the-art results on speech recognition, translation, emotion recognition, voice generation, and full-duplex dialogue while preserving the underlying...
SLAM-Omni: Timbre-Controllable Voice Interaction System with Single-Stage Training
eess.AS 2024-12 conditional novelty 5.0 of 10

A 0.5B spoken dialogue model trained end-to-end in one stage, with grouped semantic tokens for faster generation and text-only history for multi-turn dialogue.
Speech Discrete Tokens or Continuous Features? A Comparative Analysis for Spoken Language Understanding in SpeechLLMs
cs.CL 2025-08 unverdicted novelty 4.0 of 10

Under matched settings, continuous SSL speech features generally outperform discrete tokens on six spoken language understanding tasks in SpeechLLMs.
Breaking the Barriers of Text-Hungry and Audio-Deficient AI
cs.SD 2025-06 reject novelty 4.0 of 10

A proposed audio-native translation framework called MAST with fractional diffusion is described, but no evidence is given that it produces working translations.
Continuous Speech Tokens Makes LLMs Robust Multi-Modality Learners
cs.SD 2024-12 conditional novelty 4.0 of 10

A speech-to-speech dialogue model that predicts mel-spectrograms with flow matching, jointly trained with discrete text tokens, achieves lower WER than a discrete speech token baseline.

Reference graph

Works this paper leans on

258 extracted references · 3 canonical work pages · cited by 26 Pith papers

[213]

Freeze-omni: A smart and low latency speech-to-speech dialogue model with frozen llm, 2024

Xiong Wang, Yangze Li, Chaoyou Fu, Lei Xie, Ke Li, Xing Sun, and Long Ma. Freeze-omni: A smart and low latency speech-to-speech dialogue model with frozen llm, 2024

2024
[214]

Freeze-omni: A smart and low latency speech-to-speech dialogue model with frozen llm

Xiong Wang, Yangze Li, Chaoyou Fu, Lei Xie, Ke Li, Xing Sun, and Long Ma. Freeze-omni: A smart and low latency speech-to-speech dialogue model with frozen llm. arXiv preprint arXiv:2411.00774, 2024

arXiv 2024
[1]

Gpt-4 technical report

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

arXiv 2023
[2]

Musiclm: Generating music from text

Andrea Agostinelli, Timo I Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al. Musiclm: Generating music from text. arXiv preprint arXiv:2301.11325, 2023

arXiv 2023
[3]

Hilcodec: High fidelity and lightweight neural audio codec

Sunghwan Ahn, Beom Jun Woo, Min Hyun Han, Chanyeong Moon, and Nam Soo Kim. Hilcodec: High fidelity and lightweight neural audio codec. arXiv preprint arXiv:2405.04752, 2024

arXiv 2024
[4]

Apcodec: A neural audio codec with parallel amplitude and phase spectrum encoding and decoding

Yang Ai, Xiao-Hang Jiang, Ye-Xin Lu, Hui-Peng Du, and Zhen-Hua Ling. Apcodec: A neural audio codec with parallel amplitude and phase spectrum encoding and decoding. arXiv preprint arXiv:2402.10533, 2024

arXiv 2024
[5]

Seed-tts: A family of high-quality versatile speech generation models

Philip Anastassiou, Jiawei Chen, Jitong Chen, Yuanzhe Chen, Zhuo Chen, Ziyi Chen, Jian Cong, Lelai Deng, Chuang Ding, Lu Gao, et al. Seed-tts: A family of high-quality versatile speech generation models. arXiv preprint arXiv:2406.02430, 2024

arXiv 2024
[6]

Palm 2 technical report

Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023

arXiv 2023

Show all 258 references

[7]

Sd-eval: A benchmark dataset for spoken dialogue understand- ing beyond words

Junyi Ao, Yuancheng Wang, Xiaohai Tian, Dekun Chen, Jun Zhang, Lu Lu, Yuxuan Wang, Haizhou Li, and Zhizheng Wu. Sd-eval: A benchmark dataset for spoken dialogue understand- ing beyond words. arXiv preprint arXiv:2406.13340, 2024

2024 arXiv
[8]

Common voice: A massively-multilingual speech corpus

Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber. Common voice: A massively-multilingual speech corpus. arXiv preprint arXiv:1912.06670, 2019

1912 arXiv
[9]

Xls-r: Self-supervised cross-lingual speech representation learning at scale

Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick V on Platen, Yatharth Saraf, Juan Pino, et al. Xls-r: Self-supervised cross-lingual speech representation learning at scale. arXiv preprint arXiv:2111.09296, 2021. 42

2021 arXiv
[10]

wav2vec 2.0: A framework for self-supervised learning of speech representations

Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33:12449–12460, 2020

2020
[11]

Qwen technical report

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023

2023 arXiv
[12]

An empirical evaluation of generic convo- lutional and recurrent networks for sequence modeling

Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convo- lutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271, 2018

2018 arXiv
[13]

Meteor: An automatic metric for mt evaluation with improved correlation with human judgments

Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72, 2005

2005
[14]

Seamless: Multilingual expressive and streaming speech translation

Loïc Barrault, Yu-An Chung, Mariano Coria Meglioli, David Dale, Ning Dong, Mark Dup- penthaler, Paul-Ambroise Duquenne, Brian Ellis, Hady Elsahar, Justin Haaheim, et al. Seamless: Multilingual expressive and streaming speech translation. arXiv preprint arXiv:2312.05187, 2023

2023 arXiv
[15]

Curriculum learning

Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pages 41–48, 2009

2009
[16]

Medleydb: A multitrack dataset for annotation-intensive mir research

Rachel M Bittner, Justin Salamon, Mike Tierney, Matthias Mauch, Chris Cannam, and Juan Pablo Bello. Medleydb: A multitrack dataset for annotation-intensive mir research. In ISMIR, volume 14, pages 155–160, 2014

2014
[17]

A comparison of sound segregation techniques for predominant instrument recognition in musical audio signals

Juan J Bosch, Jordi Janer, Ferdinand Fuhrmann, and Perfecto Herrera. A comparison of sound segregation techniques for predominant instrument recognition in musical audio signals. In ISMIR, pages 559–564, 2012

2012
[18]

Aishell-1: An open-source mandarin speech corpus and a speech recognition baseline

Hui Bu, Jiayu Du, Xingyu Na, Bengu Wu, and Hao Zheng. Aishell-1: An open-source mandarin speech corpus and a speech recognition baseline. In 2017 20th conference of the oriental chapter of the international coordinating committee on speech databases and speech I/O systems and ...

2017
[19]

Iemocap: Interactive emotional dyadic motion capture database

Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. Iemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42:335–359, 2008

2008
[20]

Msp-improv: An acted corpus of dyadic interactions to study emotion perception

Carlos Busso, Srinivas Parthasarathy, Alec Burmania, Mohammed AbdelWahab, Najmeh Sadoughi, and Emily Mower Provost. Msp-improv: An acted corpus of dyadic interactions to study emotion perception. IEEE Transactions on Affective Computing, 8(1):67–80, 2016

2016
[21]

Xtts: a massively multilingual zero-shot text-to-speech model

Edresson Casanova, Kelly Davis, Eren Gölge, Görkem Göknar, Iulian Gulea, Logan Hart, Aya Aljafari, Joshua Meyer, Reuben Morais, Samuel Olayemi, et al. Xtts: a massively multilingual zero-shot text-to-speech model. arXiv preprint arXiv:2406.04904, 2024

2024 arXiv
[22]

Albert q

Devendra Singh Chaplot. Albert q. jiang, alexandre sablayrolles, arthur mensch, chris bamford, devendra singh chaplot, diego de las casas, florian bressand, gianna lengyel, guillaume lample, lucile saulnier, lélio renard lavaud, marie-anne lachaux, pierre stock, teven le scao,...

arXiv
[23]

Enhancing zero-shot text-to-speech synthesis with human feedback

Chen Chen, Yuchen Hu, Wen Wu, Helin Wang, Eng Siong Chng, and Chao Zhang. Enhancing zero-shot text-to-speech synthesis with human feedback. arXiv preprint arXiv:2406.00654, 2024

2024 arXiv
[24]

Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio

Guoguo Chen, Shuzhou Chai, Guanbo Wang, Jiayu Du, Wei-Qiang Zhang, Chao Weng, Dan Su, Daniel Povey, Jan Trmal, Junbo Zhang, et al. Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio. arXiv preprint arXiv:2106.06909, 2021. 43

2021 arXiv
[25]

Emova: Empowering language models to see, hear and speak with vivid emotions

Kai Chen, Yunhao Gou, Runhui Huang, Zhili Liu, Daxin Tan, Jing Xu, Chunwei Wang, Yi Zhu, Yihan Zeng, Kuo Yang, et al. Emova: Empowering language models to see, hear and speak with vivid emotions. arXiv preprint arXiv:2409.18042, 2024

2024 arXiv
[26]

Evaluating large language models trained on code

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021

2021 arXiv
[27]

Wavlm: Large-scale self-supervised pre-training for full stack speech processing

Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. Wavlm: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 16(6):...

2022
[28]

Beats: Audio pre-training with acoustic tokenizers

Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Daniel Tompkins, Zhuo Chen, and Furu Wei. Beats: Audio pre-training with acoustic tokenizers. arXiv preprint arXiv:2212.09058, 2022

2022 arXiv
[29]

V oicebench: Benchmarking llm-based voice assistants

Yiming Chen, Xianghu Yue, Chen Zhang, Xiaoxue Gao, Robby T Tan, and Haizhou Li. V oicebench: Benchmarking llm-based voice assistants. arXiv preprint arXiv:2410.17196, 2024

2024 arXiv
[30]

F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching

Yushen Chen, Zhikang Niu, Ziyang Ma, Keqi Deng, Chunhui Wang, Jian Zhao, Kai Yu, and Xie Chen. F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching. arXiv preprint arXiv:2410.06885, 2024

2024 arXiv
[31]

Audio albert: A lite bert for self-supervised learning of audio representation

Po-Han Chi, Pei-Hung Chung, Tsung-Han Wu, Chun-Cheng Hsieh, Yen-Hao Chen, Shang- Wen Li, and Hung-yi Lee. Audio albert: A lite bert for self-supervised learning of audio representation. In 2021 IEEE Spoken Language Technology Workshop (SLT), pages 344–350. IEEE, 2021

2021
[32]

Articulatory encodec: V ocal tract kinematics as a codec for speech

Cheol Jun Cho, Peter Wu, Tejas S Prabhune, Dhruv Agarwal, and Gopala K Anumanchipalli. Articulatory encodec: V ocal tract kinematics as a codec for speech. arXiv preprint arXiv:2406.12998, 2024

2024 arXiv
[33]

Qwen2-audio technical report

Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuan- jun Lv, Jinzheng He, Junyang Lin, et al. Qwen2-audio technical report. arXiv preprint arXiv:2407.10759, 2024

2024 arXiv
[34]

Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models

Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07919, 2023

2023 arXiv
[35]

Vector-quantized autoregressive predictive coding

Yu-An Chung, Hao Tang, and James Glass. Vector-quantized autoregressive predictive coding. arXiv preprint arXiv:2005.08392, 2020

2005 arXiv
[36]

Musicrl: Aligning music generation to human preferences

Geoffrey Cideron, Sertan Girgin, Mauro Verzetti, Damien Vincent, Matej Kastelic, Zalán Borsos, Brian McWilliams, Victor Ungureanu, Olivier Bachem, Olivier Pietquin, et al. Musicrl: Aligning music generation to human preferences. arXiv preprint arXiv:2402.04229, 2024

2024 arXiv
[37]

The fisher corpus: A resource for the next generations of speech-to-text

Christopher Cieri, David Miller, and Kevin Walker. The fisher corpus: A resource for the next generations of speech-to-text. In LREC, volume 4, pages 69–71, 2004

2004
[38]

Think you have solved question answering? try arc, the ai2 reasoning challenge

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

2018 arXiv
[39]

Training verifiers to solve math word problems

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

arXiv
[40]

Simple and controllable music generation

Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez. Simple and controllable music generation. Advances in Neural Information Processing Systems, 36, 2024. 44

2024
[41]

Speechverse: A large-scale generalizable audio language model

Nilaksh Das, Saket Dingliwal, Srikanth Ronanki, Rohit Paturi, David Huang, Prashant Mathur, Jie Yuan, Dhanush Bekal, Xing Niu, Sai Muralidhar Jayanthi, et al. Speechverse: A large-scale generalizable audio language model. arXiv preprint arXiv:2405.08295, 2024

2024 arXiv
[42]

FMA: A dataset for music analysis

Michaël Defferrard, Kirell Benzi, Pierre Vandergheynst, and Xavier Bresson. FMA: A dataset for music analysis. In 18th International Society for Music Information Retrieval Conference (ISMIR), 2017

2017
[43]

High fidelity neural audio compression

Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. arXiv preprint arXiv:2210.13438, 2022

2022 arXiv
[44]

Moshi: a speech-text foundation model for real-time dialogue

Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: a speech-text foundation model for real-time dialogue. arXiv preprint arXiv:2410.00037, 2024

arXiv
[45]

Bert: Pre-training of deep bidirectional transformers for language understanding

Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018

2018 arXiv
[46]

Enhancing chat language models by scaling high-quality instructional conversations

Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint arXiv:2305.14233, 2023

2023 arXiv
[47]

Attacks, defenses and evaluations for llm conversation safety: A survey

Zhichen Dong, Zhanhui Zhou, Chao Yang, Jing Shao, and Yu Qiao. Attacks, defenses and evaluations for llm conversation safety: A survey. arXiv preprint arXiv:2402.09283, 2024

2024 arXiv
[48]

Aishell-2: Transforming mandarin asr research into industrial scale

Jiayu Du, Xingyu Na, Xuechen Liu, and Hui Bu. Aishell-2: Transforming mandarin asr research into industrial scale. arXiv preprint arXiv:1808.10583, 2018

2018 arXiv
[49]

Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens

Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, et al. Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens. arXiv preprint arXiv:2407.05407, 2024

2024 arXiv
[50]

Lauragpt: Listen, attend, understand, and regenerate audio with gpt

Zhihao Du, Jiaming Wang, Qian Chen, Yunfei Chu, Zhifu Gao, Zerui Li, Kai Hu, Xiaohuan Zhou, Jin Xu, Ziyang Ma, et al. Lauragpt: Listen, attend, understand, and regenerate audio with gpt. arXiv preprint arXiv:2310.04673, 2023

2023 arXiv
[51]

Funcodec: A fundamental, reproducible and integrable open-source toolkit for neural speech codec

Zhihao Du, Shiliang Zhang, Kai Hu, and Siqi Zheng. Funcodec: A fundamental, reproducible and integrable open-source toolkit for neural speech codec. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 591–

2024
[52]

The llama 3 herd of models

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

2024 arXiv
[53]

Some signals and rules for taking speaking turns in conversations

Starkey Duncan. Some signals and rules for taking speaking turns in conversations. Journal of personality and social psychology, 23(2):283, 1972

1972
[54]

On signalling that it’s your turn to speak

Starkey Duncan Jr and George Niederehe. On signalling that it’s your turn to speak. Journal of experimental social psychology, 10(3):234–247, 1974

1974
[55]

Turngpt: a transformer-based language model for predicting turn-taking in spoken dialog

Erik Ekstedt and Gabriel Skantze. Turngpt: a transformer-based language model for predicting turn-taking in spoken dialog. arXiv preprint arXiv:2010.10874, 2020

2010 arXiv
[56]

Neural audio synthesis of musical notes with wavenet autoencoders

Jesse Engel, Cinjon Resnick, Adam Roberts, Sander Dieleman, Mohammad Norouzi, Douglas Eck, and Karen Simonyan. Neural audio synthesis of musical notes with wavenet autoencoders. In International Conference on Machine Learning, pages 1068–1077. PMLR, 2017

2017
[57]

Llama-omni: Seamless speech interaction with large language models

Qingkai Fang, Shoutao Guo, Yan Zhou, Zhengrui Ma, Shaolei Zhang, and Yang Feng. Llama-omni: Seamless speech interaction with large language models. arXiv preprint arXiv:2409.06666, 2024

2024 arXiv
[58]

Mmdialog: A large-scale multi-turn dialogue dataset towards multi-modal open-domain conversation

Jiazhan Feng, Qingfeng Sun, Can Xu, Pu Zhao, Yaming Yang, Chongyang Tao, Dongyan Zhao, and Qingwei Lin. Mmdialog: A large-scale multi-turn dialogue dataset towards multi-modal open-domain conversation. arXiv preprint arXiv:2211.05719, 2022. 45

2022 arXiv
[59]

Meisd: A mul- timodal multi-label emotion, intensity and sentiment dialogue dataset for emotion recognition and sentiment analysis in conversations

Mauajama Firdaus, Hardik Chauhan, Asif Ekbal, and Pushpak Bhattacharyya. Meisd: A mul- timodal multi-label emotion, intensity and sentiment dialogue dataset for emotion recognition and sentiment analysis in conversations. In Proceedings of the 28th international conference on ...

2020
[60]

Fsd50k: an open dataset of human-labeled sound events

Eduardo Fonseca, Xavier Favory, Jordi Pons, Frederic Font, and Xavier Serra. Fsd50k: an open dataset of human-labeled sound events. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:829–852, 2021

2021
[61]

Vita: Towards open-source interactive omni multimodal llm

Chaoyou Fu, Haojia Lin, Zuwei Long, Yunhang Shen, Meng Zhao, Yifan Zhang, Xiong Wang, Di Yin, Long Ma, Xiawu Zheng, et al. Vita: Towards open-source interactive omni multimodal llm. arXiv preprint arXiv:2408.05211, 2024

2024 arXiv
[62]

A new algorithm for data compression

Philip Gage. A new algorithm for data compression. The C Users Journal, 12(2):23–38, 1994

1994
[63]

The people’s speech: A large-scale diverse english speech recognition dataset for commercial usage

Daniel Galvez, Greg Diamos, Juan Ciro, Juan Felipe Cerón, Keith Achorn, Anjali Gopi, David Kanter, Maximilian Lam, Mark Mazumder, and Vijay Janapa Reddi. The people’s speech: A large-scale diverse english speech recognition dataset for commercial usage. arXiv preprint arXiv:21...

2021 arXiv
[64]

Augmentation invariant discrete representation for generative spoken language modeling

Itai Gat, Felix Kreuk, Tu Anh Nguyen, Ann Lee, Jade Copet, Gabriel Synnaeve, Emmanuel Dupoux, and Yossi Adi. Augmentation invariant discrete representation for generative spoken language modeling. arXiv preprint arXiv:2209.15483, 2022

2022 arXiv
[65]

Audio set: An ontology and human-labeled dataset for audio events

Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (...

2017
[66]

Audio dialogues: Dialogues dataset for audio and music understanding

Arushi Goel, Zhifeng Kong, Rafael Valle, and Bryan Catanzaro. Audio dialogues: Dialogues dataset for audio and music understanding. arXiv preprint arXiv:2404.07616, 2024

2024 arXiv
[67]

Joint audio and speech understanding

Yuan Gong, Alexander H Liu, Hongyin Luo, Leonid Karlinsky, and James Glass. Joint audio and speech understanding. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 1–8. IEEE, 2023

2023
[68]

Listen, think, and understand

Yuan Gong, Hongyin Luo, Alexander H Liu, Leonid Karlinsky, and James Glass. Listen, think, and understand. arXiv preprint arXiv:2305.10790, 2023

2023 arXiv
[69]

Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks

Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning , pages 369–376, 2006

2006
[70]

Socodec: A semantic-ordered multi-stream speech codec for efficient language model based text-to-speech synthesis

Haohan Guo, Fenglong Xie, Kun Xie, Dongchao Yang, Dake Guo, Xixin Wu, and Helen Meng. Socodec: A semantic-ordered multi-stream speech codec for efficient language model based text-to-speech synthesis. arXiv preprint arXiv:2409.00933, 2024

2024 arXiv
[71]

Prompttts: Controllable text-to-speech with text descriptions

Zhifang Guo, Yichong Leng, Yihan Wu, Sheng Zhao, and Xu Tan. Prompttts: Controllable text-to-speech with text descriptions. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023

2023
[72]

Prediction of turn-taking using multitask learning with prediction of backchannels and fillers

Kohei Hara, Koji Inoue, Katsuya Takanashi, and Tatsuya Kawahara. Prediction of turn-taking using multitask learning with prediction of backchannels and fillers. Listener, 162:364, 2018

2018
[73]

Turn-taking prediction based on detection of transition relevance place

Kohei Hara, Koji Inoue, Katsuya Takanashi, and Tatsuya Kawahara. Turn-taking prediction based on detection of transition relevance place. In INTERSPEECH, pages 4170–4174, 2019

2019
[74]

Textually pretrained speech language models

Michael Hassid, Tal Remez, Tu Anh Nguyen, Itai Gat, Alexis Conneau, Felix Kreuk, Jade Copet, Alexandre Defossez, Gabriel Synnaeve, Emmanuel Dupoux, et al. Textually pretrained speech language models. Advances in Neural Information Processing Systems, 36, 2024

2024
[75]

Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation

Haorui He, Zengqiang Shang, Chaoren Wang, Xuyuan Li, Yicheng Gu, Hua Hua, Liwei Liu, Chen Yang, Jiaqi Li, Peiyang Shi, et al. Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation. arXiv preprint arXiv:2407.05361, 2024. 46

2024 arXiv
[76]

Measuring massive multitask language understanding

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020

2009 arXiv
[77]

Cnn architectures for large-scale audio classification

Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Chan- ning Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al. Cnn architectures for large-scale audio classification. In 2017 ieee international conference on acoustics, speec...

2017
[78]

Clipscore: A reference-free evaluation metric for image captioning

Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.08718, 2021

2021 arXiv
[79]

Hubert: Self-supervised speech representation learning by masked prediction of hidden units

Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdi- nov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM transactions on audio, speech, and language processing, 2...

2021
[80]

Lora: Low-rank adaptation of large language models

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

2021 arXiv
[81]

Wavllm: Towards robust and adaptive speech large language model

Shujie Hu, Long Zhou, Shujie Liu, Sanyuan Chen, Lingwei Meng, Hongkun Hao, Jing Pan, Xunying Liu, Jinyu Li, Sunit Sivasankaran, et al. Wavllm: Towards robust and adaptive speech large language model. arXiv preprint arXiv:2404.00656, 2024

2024 arXiv
[82]

Make-an-audio 2: Temporal-enhanced text-to-audio generation

Jiawei Huang, Yi Ren, Rongjie Huang, Dongchao Yang, Zhenhui Ye, Chen Zhang, Jinglin Liu, Xiang Yin, Zejun Ma, and Zhou Zhao. Make-an-audio 2: Temporal-enhanced text-to-audio generation. arXiv preprint arXiv:2305.18474, 2023

2023 arXiv
[83]

A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions

Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qian- glong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv preprint arXiv:2311.05...

2023 arXiv
[84]

Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models

Rongjie Huang, Jiawei Huang, Dongchao Yang, Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiang Yin, and Zhou Zhao. Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models. In International Conference on Machine Learning, pages 13916–13932. PMLR, 2023

2023
[85]

Audiogpt: Understanding and generating speech, music, sound, and talking head

Rongjie Huang, Mingze Li, Dongchao Yang, Jiatong Shi, Xuankai Chang, Zhenhui Ye, Yuning Wu, Zhiqing Hong, Jiawei Huang, Jinglin Liu, et al. Audiogpt: Understanding and generating speech, music, sound, and talking head. In Proceedings of the AAAI Conference on Artificial Intell...

2024
[86]

Spiral: Self- supervised perturbation-invariant representation learning for speech pre-training

Wenyong Huang, Zhenhe Zhang, Yu Ting Yeung, Xin Jiang, and Qun Liu. Spiral: Self- supervised perturbation-invariant representation learning for speech pre-training. arXiv preprint arXiv:2201.10207, 2022

2022 arXiv
[87]

Repcodec: A speech representation codec for speech tokenization

Zhichao Huang, Chutong Meng, and Tom Ko. Repcodec: A speech representation codec for speech tokenization. arXiv preprint arXiv:2309.00169, 2023

2023 arXiv
[88]

Residual quantization with implicit neural codebooks

Iris AM Huijben, Matthijs Douze, Matthew Muckley, Ruud JG van Sloun, and Jakob Verbeek. Residual quantization with implicit neural codebooks. arXiv preprint arXiv:2401.14732, 2024

2024 arXiv
[89]

The lj speech dataset

Keith Ito and Linda Johnson. The lj speech dataset. https://keithito.com/ LJ-Speech-Dataset/, 2017

2017
[90]

Language-codec: Reducing the gaps between discrete codec representation and speech language models

Shengpeng Ji, Minghui Fang, Ziyue Jiang, Rongjie Huang, Jialung Zuo, Shulei Wang, and Zhou Zhao. Language-codec: Reducing the gaps between discrete codec representation and speech language models. arXiv preprint arXiv:2402.12208, 2024

2024 arXiv
[91]

Mobilespeech: A fast and high-fidelity framework for mobile zero-shot text-to-speech

Shengpeng Ji, Ziyue Jiang, Hanting Wang, Jialong Zuo, and Zhou Zhao. Mobilespeech: A fast and high-fidelity framework for mobile zero-shot text-to-speech. arXiv preprint arXiv:2402.09378, 2024. 47

2024 arXiv
[92]

Wavtokenizer: an efficient acoustic discrete codec tokenizer for audio language modeling

Shengpeng Ji, Ziyue Jiang, Wen Wang, Yifu Chen, Minghui Fang, Jialong Zuo, Qian Yang, Xize Cheng, Zehan Wang, Ruiqi Li, et al. Wavtokenizer: an efficient acoustic discrete codec tokenizer for audio language modeling. arXiv preprint arXiv:2408.16532, 2024

2024 arXiv
[93]

Textrolspeech: A text style control speech corpus with codec language text-to-speech models

Shengpeng Ji, Jialong Zuo, Minghui Fang, Ziyue Jiang, Feiyang Chen, Xinyu Duan, Baoxing Huai, and Zhou Zhao. Textrolspeech: A text style control speech corpus with codec language text-to-speech models. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and ...

2024
[94]

Controlspeech: Towards simultaneous zero-shot speaker cloning and zero-shot language style control with decoupled codec

Shengpeng Ji, Jialong Zuo, Minghui Fang, Siqi Zheng, Qian Chen, Wen Wang, Ziyue Jiang, Hai Huang, Xize Cheng, Rongjie Huang, et al. Controlspeech: Towards simultaneous zero-shot speaker cloning and zero-shot language style control with decoupled codec. arXiv preprint arXiv:240...

2024 arXiv
[95]

Cvss corpus and massively multilingual speech-to-speech translation

Ye Jia, Michelle Tadmor Ramanovich, Quan Wang, and Heiga Zen. Cvss corpus and massively multilingual speech-to-speech translation. arXiv preprint arXiv:2201.03713, 2022

2022 arXiv
[96]

Mixtral of experts

Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024

2024 arXiv
[97]

Mega-tts 2: Boosting prompting mechanisms for zero-shot speech synthesis

Ziyue Jiang, Jinglin Liu, Yi Ren, Jinzheng He, Zhenhui Ye, Shengpeng Ji, Qian Yang, Chen Zhang, Pengfei Wei, Chunfeng Wang, et al. Mega-tts 2: Boosting prompting mechanisms for zero-shot speech synthesis. In The Twelfth International Conference on Learning Representa- tions, 2024

2024
[98]

Mega-tts: Zero-shot text-to-speech at scale with intrinsic inductive bias

Ziyue Jiang, Yi Ren, Zhenhui Ye, Jinglin Liu, Chen Zhang, Qian Yang, Shengpeng Ji, Rongjie Huang, Chunfeng Wang, Xiang Yin, et al. Mega-tts: Zero-shot text-to-speech at scale with intrinsic inductive bias. arXiv preprint arXiv:2306.03509, 2023

2023 arXiv
[99]

Duplex conversation in outbound agent system

Chunxiang Jin, Minghui Yang, and Zujie Wen. Duplex conversation in outbound agent system. In Interspeech, pages 4866–4867, 2021

2021
[100]

Efficient multimodal large language models: A survey

Yizhang Jin, Jian Li, Yexin Liu, Tianjun Gu, Kai Wu, Zhengkai Jiang, Muyang He, Bo Zhao, Xin Tan, Zhenye Gan, et al. Efficient multimodal large language models: A survey. arXiv preprint arXiv:2405.10739, 2024

2024
[101]

Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models

Zeqian Ju, Yuancheng Wang, Kai Shen, Xu Tan, Detai Xin, Dongchao Yang, Yanqing Liu, Yichong Leng, Kaitao Song, Siliang Tang, et al. Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models. arXiv preprint arXiv:2403.03100, 2024

2024 arXiv
[102]

Libri-light: A benchmark for asr with limited or no supervision

Jacob Kahn, Morgane Riviere, Weiyi Zheng, Evgeny Kharitonov, Qiantong Xu, Pierre- Emmanuel Mazaré, Julien Karadayi, Vitaliy Liptchinsky, Ronan Collobert, Christian Fuegen, et al. Libri-light: A benchmark for asr with limited or no supervision. In ICASSP 2020-2020 IEEE Internat...

2020
[103]

textless-lib: A library for textless spoken language processing

Eugene Kharitonov, Jade Copet, Kushal Lakhotia, Tu Anh Nguyen, Paden Tomasello, Ann Lee, Ali Elkahky, Wei-Ning Hsu, Abdelrahman Mohamed, Emmanuel Dupoux, et al. textless-lib: A library for textless spoken language processing. arXiv preprint arXiv:2202.07359, 2022

2022 arXiv
[104]

Reinforcement learning for turn- taking management in incremental spoken dialogue systems

Hatim Khouzaimi, Romain Laroche, and Fabrice Lefèvre. Reinforcement learning for turn- taking management in incremental spoken dialogue systems. In IJCAI, pages 2831–2837, 2016

2016
[105]

Fr \’echet au- dio distance: A metric for evaluating music enhancement algorithms

Kevin Kilgour, Mauricio Zuluaga, Dominik Roblek, and Matthew Sharifi. Fr \’echet au- dio distance: A metric for evaluating music enhancement algorithms. arXiv preprint arXiv:1812.08466, 2018

2018 arXiv
[106]

Audiocaps: Gen- erating captions for audios in the wild

Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. Audiocaps: Gen- erating captions for audios in the wild. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume...

2019
[107]

Unified speech-text pretraining for spoken dialog modeling

Heeseung Kim, Soonshin Seo, Kyeongseok Jeong, Ohsung Kwon, Jungwhan Kim, Jaehong Lee, Eunwoo Song, Myungwoo Oh, Sungroh Yoon, and Kang Min Yoo. Unified speech-text pretraining for spoken dialog modeling. arXiv preprint arXiv:2402.05706, 2024

2024 arXiv
[108]

Conditional variational autoencoder with ad- versarial learning for end-to-end text-to-speech

Jaehyeon Kim, Jungil Kong, and Juhee Son. Conditional variational autoencoder with ad- versarial learning for end-to-end text-to-speech. In International Conference on Machine Learning, pages 5530–5540. PMLR, 2021

2021
[109]

Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis

Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in neural information processing systems, 33:17022–17033, 2020

2020
[110]

Vits2: Improving quality and efficiency of single-stage text-to-speech with adversarial learning and architecture design

Jungil Kong, Jihoon Park, Beomjeong Kim, Jeongmin Kim, Dohee Kong, and Sangjin Kim. Vits2: Improving quality and efficiency of single-stage text-to-speech with adversarial learning and architecture design. arXiv preprint arXiv:2307.16430, 2023

2023 arXiv
[111]

Panns: Large-scale pretrained audio neural networks for audio pattern recognition

Qiuqiang Kong, Yin Cao, Turab Iqbal, Yuxuan Wang, Wenwu Wang, and Mark D Plumbley. Panns: Large-scale pretrained audio neural networks for audio pattern recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:2880–2894, 2020

2020
[112]

Audio flamingo: A novel audio language model with few-shot learning and dialogue abilities

Zhifeng Kong, Arushi Goel, Rohan Badlani, Wei Ping, Rafael Valle, and Bryan Catanzaro. Audio flamingo: A novel audio language model with few-shot learning and dialogue abilities. arXiv preprint arXiv:2402.01831, 2024

2024 arXiv
[113]

Openassistant conversations-democratizing large language model alignment.Advances in Neural Information Processing Systems, 36, 2024

Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Nguyen, Oliver Stanley, Richárd Nagyfi, et al. Openassistant conversations-democratizing large language model alignment.Advances in Neural Information Proce...

2024
[114]

High-fidelity audio compression with improved rvqgan

Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High-fidelity audio compression with improved rvqgan. Advances in Neural Information Processing Systems, 36, 2024

2024
[115]

Smooth turn-taking by a robot using an online continuous model to generate turn-taking cues

Divesh Lala, Koji Inoue, and Tatsuya Kawahara. Smooth turn-taking by a robot using an online continuous model to generate turn-taking cues. In 2019 International Conference on Multimodal Interaction, pages 226–234, 2019

2019
[116]

Attentive listening system with backchanneling, response generation and flexible turn-taking

Divesh Lala, Pierrick Milhorat, Koji Inoue, Masanari Ishida, Katsuya Takanashi, and Tatsuya Kawahara. Attentive listening system with backchanneling, response generation and flexible turn-taking. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, page...

2017
[117]

Efficient neural music generation

Max WY Lam, Qiao Tian, Tang Li, Zongyu Yin, Siyuan Feng, Ming Tu, Yuliang Ji, Rui Xia, Mingbo Ma, Xuchen Song, et al. Efficient neural music generation. Advances in Neural Information Processing Systems, 36, 2024

2024
[118]

V oicebox: Text-guided multilingual universal speech generation at scale

Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, et al. V oicebox: Text-guided multilingual universal speech generation at scale. Advances in neural information processing systems, 36, 2024

2024
[119]

Prompttts 2: Describing and generating voices with text prompt

Yichong Leng, Zhifang Guo, Kai Shen, Xu Tan, Zeqian Ju, Yanqing Liu, Yufei Liu, Dongchao Yang, Leying Zhang, Kaitao Song, et al. Prompttts 2: Describing and generating voices with text prompt. arXiv preprint arXiv:2309.02285, 2023

2023 arXiv
[120]

Single-codec: Single-codebook speech codec towards high-performance speech generation

Hanzhao Li, Liumeng Xue, Haohan Guo, Xinfa Zhu, Yuanjun Lv, Lei Xie, Yunlin Chen, Hao Yin, and Zhifei Li. Single-codec: Single-codebook speech codec towards high-performance speech generation. arXiv preprint arXiv:2406.07422, 2024

2024 arXiv
[121]

A survey on benchmarks of multimodal large language models, 2024

Jian Li, Weiheng Lu, Hao Fei, Meng Luo, Ming Dai, Min Xia, Yizhang Jin, Zhenye Gan, Ding Qi, Chaoyou Fu, Ying Tai, Wankou Yang, Yabiao Wang, and Chengjie Wang. A survey on benchmarks of multimodal large language models, 2024. 49

2024
[122]

Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730–19742. PMLR, 2023

2023
[123]

Baichuan-omni technical report

Yadong Li, Haoze Sun, Mingan Lin, Tianpeng Li, Guosheng Dong, Tao Zhang, Bowen Ding, Wei Song, Zhenglin Cheng, Yuqi Huo, et al. Baichuan-omni technical report. arXiv preprint arXiv:2410.08565, 2024

2024
[124]

Exploration of a self- supervised speech model: A study on emotional corpora

Yuanchao Li, Yumnah Mohamied, Peter Bell, and Catherine Lai. Exploration of a self- supervised speech model: A study on emotional corpora. In 2022 IEEE Spoken Language Technology Workshop (SLT), pages 868–875. IEEE, 2023

2022
[125]

Openorca: An open dataset of gpt augmented flan reasoning traces

Wing Lian, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet V ong, and "Teknium". Openorca: An open dataset of gpt augmented flan reasoning traces. https://https:// huggingface.co/Open-Orca/OpenOrca, 2023

2023
[126]

Holistic evaluation of language models

Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110, 2022

2022 arXiv
[127]

Rouge: A package for automatic evaluation of summaries

Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summariza- tion branches out, pages 74–81, 2004

2004
[128]

Advancing large language models to capture varied speaking styles and respond properly in spoken conversations

Guan-Ting Lin, Cheng-Han Chiang, and Hung-yi Lee. Advancing large language models to capture varied speaking styles and respond properly in spoken conversations. arXiv preprint arXiv:2402.12786, 2024

2024 arXiv
[129]

Paralinguistics-enhanced large language modeling of spoken dialogue

Guan-Ting Lin, Prashanth Gurunath Shivakumar, Ankur Gandhe, Chao-Han Huck Yang, Yile Gu, Shalini Ghosh, Andreas Stolcke, Hung-yi Lee, and Ivan Bulyko. Paralinguistics-enhanced large language modeling of spoken dialogue. In ICASSP 2024-2024 IEEE International Conference on Acou...

2024
[130]

Align-slm: Textless spoken language models with reinforce- ment learning from ai feedback, 2024

Guan-Ting Lin, Prashanth Gurunath Shivakumar, Aditya Gourav, Yile Gu, Ankur Gandhe, Hung yi Lee, and Ivan Bulyko. Align-slm: Textless spoken language models with reinforce- ment learning from ai feedback, 2024

2024
[131]

Duplex conversation: Towards human-like interaction in spoken dialogue systems

Ting-En Lin, Yuchuan Wu, Fei Huang, Luo Si, Jian Sun, and Yongbin Li. Duplex conversation: Towards human-like interaction in spoken dialogue systems. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 3299–3308, 2022

2022
[132]

Flow matching for generative modeling

Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022

2022 arXiv
[133]

Clotho-aqa: A crowdsourced dataset for audio question answering

Samuel Lipping, Parthasaarathy Sudarsanam, Konstantinos Drossos, and Tuomas Virtanen. Clotho-aqa: A crowdsourced dataset for audio question answering. In 2022 30th European Signal Processing Conference (EUSIPCO), pages 1140–1144. IEEE, 2022

2022
[134]

Mockingjay: Unsupervised speech representation learning with deep bidirectional transformer encoders

Andy T Liu, Shu-wen Yang, Po-Han Chi, Po-chun Hsu, and Hung-yi Lee. Mockingjay: Unsupervised speech representation learning with deep bidirectional transformer encoders. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Process- ing (ICASSP), pa...

2020
[135]

Audioldm: Text-to-audio generation with latent diffusion models

Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. Audioldm: Text-to-audio generation with latent diffusion models. arXiv preprint arXiv:2301.12503, 2023

2023 arXiv
[136]

Semanticodec: An ultra low bitrate semantic audio codec for general sound

Haohe Liu, Xuenan Xu, Yi Yuan, Mengyue Wu, Wenwu Wang, and Mark D Plumbley. Semanticodec: An ultra low bitrate semantic audio codec for general sound. arXiv preprint arXiv:2405.00233, 2024

2024 arXiv
[137]

Audioldm 2: Learning holistic audio generation with self-supervised pretraining

Haohe Liu, Yi Yuan, Xubo Liu, Xinhao Mei, Qiuqiang Kong, Qiao Tian, Yuping Wang, Wenwu Wang, Yuxuan Wang, and Mark D Plumbley. Audioldm 2: Learning holistic audio generation with self-supervised pretraining. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024. 50

2024
[138]

Generative expressive conversational speech synthesis

Rui Liu, Yifan Hu, Ren Yi, Yin Xiang, and Haizhou Li. Generative expressive conversational speech synthesis. arXiv preprint arXiv:2407.21491, 2024

2024 arXiv
[139]

G-eval: Nlg evaluation using gpt-4 with better human alignment

Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303.16634, 2023

2023 arXiv
[140]

The flan collection: Designing data and methods for effective instruction tuning

Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. In International Conference on Machine Learning , pages 22631–22648. P...

2023
[141]

Natural language guidance of high-fidelity text-to-speech with synthetic annotations

Dan Lyth and Simon King. Natural language guidance of high-fidelity text-to-speech with synthetic annotations. arXiv preprint arXiv:2402.01912, 2024

2024 arXiv
[142]

Foundation models for music: A survey

Yinghao Ma, Anders Øland, Anton Ragni, Bleiz MacSen Del Sette, Charalampos Saitis, Chris Donahue, Chenghua Lin, Christos Plachouras, Emmanouil Benetos, Elio Quinton, et al. Foundation models for music: A survey. arXiv preprint arXiv:2408.14340, 2024

2024 arXiv
[143]

Language model can listen while speaking

Ziyang Ma, Yakun Song, Chenpeng Du, Jian Cong, Zhuo Chen, Yuping Wang, Yuxuan Wang, and Xie Chen. Language model can listen while speaking. arXiv preprint arXiv:2408.02622, 2024

2024 arXiv
[144]

emotion2vec: Self-supervised pre-training for speech emotion representation

Ziyang Ma, Zhisheng Zheng, Jiaxin Ye, Jinchao Li, Zhifu Gao, Shiliang Zhang, and Xie Chen. emotion2vec: Self-supervised pre-training for speech emotion representation. arXiv preprint arXiv:2312.15185, 2023

2023 arXiv
[145]

Alpaca: Intermittent execution without checkpoints

Kiwan Maeng, Alexei Colin, and Brandon Lucia. Alpaca: Intermittent execution without checkpoints. Proceedings of the ACM on Programming Languages, 1(OOPSLA):1–30, 2017

2017
[146]

V oxtlm: Unified decoder-only models for consolidating speech recognition, synthesis and speech, text continuation tasks

Soumi Maiti, Yifan Peng, Shukjae Choi, Jee-weon Jung, Xuankai Chang, and Shinji Watanabe. V oxtlm: Unified decoder-only models for consolidating speech recognition, synthesis and speech, text continuation tasks. In ICASSP 2024-2024 IEEE International Conference on Acoustics, S...

2024
[147]

Spoken language interaction with robots: Recommendations for future research

Matthew Marge, Carol Espy-Wilson, Nigel G Ward, Abeer Alwan, Yoav Artzi, Mohit Bansal, Gil Blankenship, Joyce Chai, Hal Daumé III, Debadeepta Dey, et al. Spoken language interaction with robots: Recommendations for future research. Computer Speech & Language, 71:101255, 2022

2022
[148]

Wavcaps: A chatgpt-assisted weakly-labelled audio captioning dataset for audio-language multimodal research

Xinhao Mei, Chutong Meng, Haohe Liu, Qiuqiang Kong, Tom Ko, Chengqi Zhao, Mark D Plumbley, Yuexian Zou, and Wenwu Wang. Wavcaps: A chatgpt-assisted weakly-labelled audio captioning dataset for audio-language multimodal research. IEEE/ACM Transactions on Audio, Speech, and Lang...

2024
[149]

Sd-gpt: Autoregressive spoken dialogue language modeling with decoder-only transformers

Ziqiao Meng, Qichao Wang, Wenqian Cui, Yifei Zhang, Bingzhe Wu, Irwin King, Liang Chen, and Peilin Zhao. Sd-gpt: Autoregressive spoken dialogue language modeling with decoder-only transformers. In Audio Imagination: NeurIPS 2024 Workshop AI-Driven Speech, Music, and Sound Generation

2024
[150]

Finite scalar quantization: Vq-vae made simple

Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantization: Vq-vae made simple. arXiv preprint arXiv:2309.15505, 2023

2023 arXiv
[151]

Dcase 2017 challenge setup: Tasks, datasets and baseline system

Annamaria Mesaros, Toni Heittola, Aleksandr Diment, Benjamin Elizalde, Ankit Shah, Em- manuel Vincent, Bhiksha Raj, and Tuomas Virtanen. Dcase 2017 challenge setup: Tasks, datasets and baseline system. In DCASE 2017-workshop on detection and classification of acoustic scenes a...

2017
[152]

Tut database for acoustic scene classification and sound event detection

Annamaria Mesaros, Toni Heittola, and Tuomas Virtanen. Tut database for acoustic scene classification and sound event detection. In 2016 24th European Signal Processing Conference (EUSIPCO), pages 1128–1132. IEEE, 2016

2016
[153]

Sound event detection: A tutorial

Annamaria Mesaros, Toni Heittola, Tuomas Virtanen, and Mark D Plumbley. Sound event detection: A tutorial. IEEE Signal Processing Magazine, 38(5):67–83, 2021. 51

2021
[154]

Cross-task general- ization via natural language crowdsourcing instructions

Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task general- ization via natural language crowdsourcing instructions. arXiv preprint arXiv:2104.08773, 2021

2021 arXiv
[155]

Pslm: Parallel generation of text and speech with llms for low-latency spoken dialogue systems

Kentaro Mitsui, Koh Mitsuda, Toshiaki Wakatsuki, Yukiya Hono, and Kei Sawada. Pslm: Parallel generation of text and speech with llms for low-latency spoken dialogue systems. arXiv preprint arXiv:2406.12428, 2024

2024 arXiv
[156]

Self- supervised speech representation learning: A review

Abdelrahman Mohamed, Hung-yi Lee, Lasse Borgholt, Jakob D Havtorn, Joakim Edin, Christian Igel, Katrin Kirchhoff, Shang-Wen Li, Karen Livescu, Lars Maaløe, et al. Self- supervised speech representation learning: A review. IEEE Journal of Selected Topics in Signal Processing, 1...

2022
[157]

Spo- ken question answering and speech continuation using spectrogram-powered llm

Eliya Nachmani, Alon Levkovitch, Roy Hirsch, Julian Salazar, Chulayuth Asawaroengchai, Soroosh Mariooryad, Ehud Rivlin, RJ Skerry-Ryan, and Michelle Tadmor Ramanovich. Spo- ken question answering and speech continuation using spectrogram-powered llm. arXiv preprint arXiv:2305....

2023 arXiv
[158]

Generative spoken dialogue language modeling

Tu Anh Nguyen, Eugene Kharitonov, Jade Copet, Yossi Adi, Wei-Ning Hsu, Ali Elkahky, Paden Tomasello, Robin Algayres, Benoit Sagot, Abdelrahman Mohamed, et al. Generative spoken dialogue language modeling. Transactions of the Association for Computational Linguistics, 11:250–266, 2023

2023
[159]

Spirit-lm: Interleaved spoken and written language model

Tu Anh Nguyen, Benjamin Muller, Bokai Yu, Marta R Costa-Jussa, Maha Elbayad, Sravya Popuri, Paul-Ambroise Duquenne, Robin Algayres, Ruslan Mavlyutov, Itai Gat, et al. Spirit-lm: Interleaved spoken and written language model. arXiv preprint arXiv:2402.05755, 2024

2024 arXiv
[160]

Cleans2s: High-quality and streaming speech-to-speech interactive agent in a single file

Yazhe Niu, Shuai Hu, and Yun Chen. Cleans2s: High-quality and streaming speech-to-speech interactive agent in a single file. https://github.com/opendilab/CleanS2S, 2024

2024
[161]

Librispeech: an asr corpus based on public domain audio books

Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5206–5210. IEEE, 2015

2015
[162]

Bleu: a method for automatic evaluation of machine translation

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. InProceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318, 2002

2002
[163]

Let’s go real talk: Spoken dialogue model for face-to-face conversation

Se Jin Park, Chae Won Kim, Hyeongseop Rha, Minsu Kim, Joanna Hong, Jeong Hun Yeo, and Yong Man Ro. Let’s go real talk: Spoken dialogue model for face-to-face conversation. arXiv preprint arXiv:2406.07867, 2024

2024 arXiv
[164]

V oice- craft: Zero-shot speech editing and text-to-speech in the wild.arXiv preprint arXiv:2403.16973, 2024

Puyuan Peng, Po-Yao Huang, Daniel Li, Abdelrahman Mohamed, and David Harwath. V oice- craft: Zero-shot speech editing and text-to-speech in the wild.arXiv preprint arXiv:2403.16973, 2024

2024 arXiv
[165]

Encodecmae: Leveraging neural codecs for universal audio representation learning

Leonardo Pepino, Pablo Riera, and Luciana Ferrer. Encodecmae: Leveraging neural codecs for universal audio representation learning. arXiv preprint arXiv:2309.07391, 2023

2023 arXiv
[166]

Esc: Dataset for environmental sound classification

Karol J Piczak. Esc: Dataset for environmental sound classification. In Proceedings of the 23rd ACM international conference on Multimedia, pages 1015–1018, 2015

2015
[167]

Speech resynthesis from discrete disentangled self-supervised representations

Adam Polyak, Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Ab- delrahman Mohamed, and Emmanuel Dupoux. Speech resynthesis from discrete disentangled self-supervised representations. arXiv preprint arXiv:2104.00355, 2021

2021 arXiv
[168]

Meld: A multimodal multi-party dataset for emotion recognition in conversations

Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. Meld: A multimodal multi-party dataset for emotion recognition in conversations. arXiv preprint arXiv:1810.02508, 2018

2018 arXiv
[169]

Mls: A large-scale multilingual dataset for speech research

Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert. Mls: A large-scale multilingual dataset for speech research. arXiv preprint arXiv:2012.03411, 2020. 52

2012 arXiv
[170]

Robust speech recognition via large-scale weak supervision

Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, pages 28492–28518. PMLR, 2023

2023
[171]

Direct preference optimization: Your language model is secretly a reward model

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024

2024
[172]

Musdb18-a corpus for music separation

Zafar Rafii, Antoine Liutkus, Fabian-Robert Stöter, Stylianos Ioannis Mimilakis, and Rachel Bittner. Musdb18-a corpus for music separation. 2017

2017
[173]

M3-audiodec: Multi-channel multi-speaker multi-spatial audio codec

Anton Ratnarajah, Shi-Xiong Zhang, and Dong Yu. M3-audiodec: Multi-channel multi-speaker multi-spatial audio codec. arXiv preprint arXiv:2309.07416, 2023

2023 arXiv
[174]

A finite-state turn-taking model for spoken dialog systems

Antoine Raux and Maxine Eskenazi. A finite-state turn-taking model for spoken dialog systems. In Proceedings of human language technologies: The 2009 annual conference of the North American chapter of the association for computational linguistics, pages 629–637, 2009

2009
[175]

The interspeech 2020 deep noise suppression challenge: Datasets, subjective speech quality and testing framework

CK Reddy, E Beyrami, H Dubey, V Gopal, R Cheng, R Cutler, S Matusevych, R Aichner, A Aazami, S Braun, et al. The interspeech 2020 deep noise suppression challenge: Datasets, subjective speech quality and testing framework. arxiv 2020. arXiv preprint arXiv:2001.08662

2020 arXiv
[176]

Coqa: A conversational question answering challenge

Siva Reddy, Danqi Chen, and Christopher D Manning. Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249– 266, 2019

2019
[177]

Fastspeech 2: Fast and high-quality end-to-end text to speech

Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. Fastspeech 2: Fast and high-quality end-to-end text to speech. arXiv preprint arXiv:2006.04558, 2020

2006 arXiv
[178]

Fewer-token neural speech codec with time-invariant codes

Yong Ren, Tao Wang, Jiangyan Yi, Le Xu, Jianhua Tao, Chu Yuan Zhang, and Junzuo Zhou. Fewer-token neural speech codec with time-invariant codes. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 12737–12741. IEEE, 2024

2024
[179]

Ted-lium: an automatic speech recognition dedicated corpus

Anthony Rousseau, Paul Deléglise, and Yannick Esteve. Ted-lium: an automatic speech recognition dedicated corpus. In LREC, pages 125–129, 2012

2012
[180]

Audiopalm: A large language model that can speak and listen

Paul K Rubenstein, Chulayuth Asawaroengchai, Duc Dung Nguyen, Ankur Bapna, Za- lán Borsos, Félix de Chaumont Quitry, Peter Chen, Dalia El Badawy, Wei Han, Eugene Kharitonov, et al. Audiopalm: A large language model that can speak and listen. arXiv preprint arXiv:2306.12925, 2023

2023 arXiv
[181]

A simplest systematics for the organization of turn-taking for conversation

Harvey Sacks, Emanuel A Schegloff, and Gail Jefferson. A simplest systematics for the organization of turn-taking for conversation. language, 50(4):696–735, 1974

1974
[182]

Winogrande: An adversarial winograd schema challenge at scale

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106, 2021

2021
[183]

Mmau: A massive multi-task audio understanding and reasoning benchmark

S Sakshi, Utkarsh Tyagi, Sonal Kumar, Ashish Seth, Ramaneswaran Selvakumar, Oriol Nieto, Ramani Duraiswami, Sreyan Ghosh, and Dinesh Manocha. Mmau: A massive multi-task audio understanding and reasoning benchmark. arXiv preprint arXiv:2410.19168, 2024

arXiv
[184]

A dataset and taxonomy for urban sound research

Justin Salamon, Christopher Jacoby, and Juan Pablo Bello. A dataset and taxonomy for urban sound research. In Proceedings of the 22nd ACM international conference on Multimedia, pages 1041–1044, 2014

2014
[185]

wav2vec: Unsuper- vised pre-training for speech recognition

Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. wav2vec: Unsuper- vised pre-training for speech recognition. arXiv preprint arXiv:1904.05862, 2019

1904 arXiv
[186]

Proximal policy optimization algorithms

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. 53

2017 arXiv
[187]

Neural machine translation of rare words with subword units

Rico Sennrich. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909, 2015

2015 arXiv
[188]

Acquiring language from speech by learning to remember and predict

Cory Shain and Micha Elsner. Acquiring language from speech by learning to remember and predict. In Proceedings of the 24th Conference on Computational Natural Language Learning, pages 195–214, 2020

2020
[189]

Low bitrate high-quality rvqgan-based discrete speech tokenizer

Slava Shechtman and Avihu Dekel. Low bitrate high-quality rvqgan-based discrete speech tokenizer. In Interspeech 2024, pages 4174–4178, 2024

2024
[190]

Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers

Kai Shen, Zeqian Ju, Xu Tan, Yanqing Liu, Yichong Leng, Lei He, Tao Qin, Sheng Zhao, and Jiang Bian. Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers. arXiv preprint arXiv:2304.09116, 2023

2023 arXiv
[191]

Aishell-3: A multi-speaker mandarin tts corpus and the baselines

Yao Shi, Hui Bu, Xin Xu, Shaoji Zhang, and Ming Li. Aishell-3: A multi-speaker mandarin tts corpus and the baselines. arXiv preprint arXiv:2010.11567, 2020

2010 arXiv
[192]

Llasm: Large language and speech model, 2023

Yu Shu, Siwei Dong, Guangyao Chen, Wenhao Huang, Ruihua Zhang, Daochen Shi, Qiqi Xiang, and Yemin Shi. Llasm: Large language and speech model, 2023

2023
[193]

Spokenwoz: A large-scale speech-text benchmark for spoken task-oriented dialogue agents

Shuzheng Si, Wentao Ma, Haoyu Gao, Yuchuan Wu, Ting-En Lin, Yinpei Dai, Hangyu Li, Rui Yan, Fei Huang, and Yongbin Li. Spokenwoz: A large-scale speech-text benchmark for spoken task-oriented dialogue agents. Advances in Neural Information Processing Systems, 36, 2024

2024
[194]

Snac: Multi-scale neural audio codec

Hubert Siuzdak, Florian Grötschla, and Luca A Lanzendörfer. Snac: Multi-scale neural audio codec. arXiv preprint arXiv:2410.14411, 2024

2024 arXiv
[195]

Musan: A music, speech, and noise corpus

David Snyder, Guoguo Chen, and Daniel Povey. Musan: A music, speech, and noise corpus. arXiv preprint arXiv:1510.08484, 2015

2015 arXiv
[196]

Funaudiollm: V oice understanding and generation foundation models for natural interaction between humans and llms

Tongyi SpeechTeam. Funaudiollm: V oice understanding and generation foundation models for natural interaction between humans and llms. arXiv preprint arXiv:2407.04051, 2024

2024 arXiv
[197]

Policy gradi- ent methods for reinforcement learning with function approximation

Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradi- ent methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999

1999
[198]

Commonsenseqa: A question answering challenge targeting commonsense knowledge

Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. arXiv preprint arXiv:1811.00937, 2018

2018 arXiv
[199]

Salmonn: Towards generic hearing abilities for large language models

Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. Salmonn: Towards generic hearing abilities for large language models. arXiv preprint arXiv:2310.13289, 2023

2023 arXiv
[200]

Kespeech: An open source speech dataset of mandarin and its eight subdialects

Zhiyuan Tang, Dong Wang, Yanguang Xu, Jianwei Sun, Xiaoning Lei, Shuaijiang Zhao, Cheng Wen, Xingjun Tan, Chuandong Xie, Shuran Zhou, et al. Kespeech: An open source speech dataset of mandarin and its eight subdialects. In Thirty-fifth Conference on Neural Information Processi...

2021
[201]

Llama: Open and efficient foundation language models

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

2023 arXiv
[202]

Attention is all you need

A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017

2017
[203]

The voice bank corpus: Design, collection and data analysis of a large regional accent speech database

Christophe Veaux, Junichi Yamagishi, and Simon King. The voice bank corpus: Design, collection and data analysis of a large regional accent speech database. In 2013 international conference oriental COCOSDA held jointly with 2013 conference on Asian spoken language research an...

2013
[204]

Beyond turn-based interfaces: Synchronous llms as full-duplex dialogue agents

Bandhav Veluri, Benjamin N Peloquin, Bokai Yu, Hongyu Gong, and Shyamnath Gollakota. Beyond turn-based interfaces: Synchronous llms as full-duplex dialogue agents. arXiv preprint arXiv:2409.15594, 2024

2024 arXiv
[205]

Diffusion model alignment using direct preference optimization

Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and...

2024
[206]

Audiobench: A universal benchmark for audio large language models

Bin Wang, Xunlong Zou, Geyu Lin, Shuo Sun, Zhuohan Liu, Wenyu Zhang, Zhengyuan Liu, AiTi Aw, and Nancy F Chen. Audiobench: A universal benchmark for audio large language models. arXiv preprint arXiv:2406.16020, 2024

2024 arXiv
[207]

Covost: A diverse multilingual speech-to-text translation corpus

Changhan Wang, Juan Pino, Anne Wu, and Jiatao Gu. Covost: A diverse multilingual speech-to-text translation corpus. arXiv preprint arXiv:2002.01320, 2020

2002 arXiv
[208]

V oxpopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation

Changhan Wang, Morgane Riviere, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux. V oxpopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation. arXiv preprint a...

2021 arXiv
[209]

Blsp: Bootstrapping language-speech pre-training via behavior alignment of continuation writing

Chen Wang, Minpeng Liao, Zhongqiang Huang, Jinliang Lu, Junhong Wu, Yuchen Liu, Chengqing Zong, and Jiajun Zhang. Blsp: Bootstrapping language-speech pre-training via behavior alignment of continuation writing. arXiv preprint arXiv:2309.00916, 2023

2023 arXiv
[210]

Neural codec language models are zero-shot text to speech synthesizers

Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301.02111, 2023

2023 arXiv
[211]

Ham-tts: Hierarchical acoustic modeling for token-based zero-shot text-to-speech with model and data scaling

Chunhui Wang, Chang Zeng, Bowen Zhang, Ziyang Ma, Yefan Zhu, Zifeng Cai, Jian Zhao, Zhonglin Jiang, and Yong Chen. Ham-tts: Hierarchical acoustic modeling for token-based zero-shot text-to-speech with model and data scaling. arXiv preprint arXiv:2403.05989, 2024

2024 arXiv
[212]

A full-duplex speech dialogue scheme based on large language models

Peng Wang, Songshuo Lu, Yaohua Tang, Sijie Yan, Yuanjun Xiong, and Wei Xia. A full-duplex speech dialogue scheme based on large language models. arXiv preprint arXiv:2405.19487, 2024

2024 arXiv
[215]

Self-instruct: Aligning language models with self-generated instruc- tions

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instruc- tions. arXiv preprint arXiv:2212.10560, 2022

2022 arXiv
[216]

Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks

Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks. arXiv pre...

2022 arXiv
[217]

Maskgct: Zero-shot text-to-speech with masked generative codec transformer

Yuancheng Wang, Haoyue Zhan, Liwei Liu, Ruihong Zeng, Haotian Guo, Jiachen Zheng, Qiang Zhang, Shunsi Zhang, and Zhizheng Wu. Maskgct: Zero-shot text-to-speech with masked generative codec transformer. arXiv preprint arXiv:2409.00750, 2024

2024 arXiv
[218]

Finetuned language models are zero-shot learners

Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021

2021 arXiv
[219]

Chain-of-thought prompting elicits reasoning in large language models

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022. 55

2022
[220]

U2++: Unified two-pass bidirectional end-to-end model for speech recognition

Di Wu, Binbin Zhang, Chao Yang, Zhendong Peng, Wenjing Xia, Xiaoyu Chen, and Xin Lei. U2++: Unified two-pass bidirectional end-to-end model for speech recognition. arXiv preprint arXiv:2106.05642, 2021

2021 arXiv
[221]

Audiodec: An open- source streaming high-fidelity neural audio codec

Yi-Chiao Wu, Israel D Gebru, Dejan Markovi´c, and Alexander Richard. Audiodec: An open- source streaming high-fidelity neural audio codec. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023

2023
[222]

Show-o: One single transformer to unify multimodal understanding and generation

Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408.12528, 2024

2024 arXiv
[223]

Mini-omni: Language models can hear, talk while thinking in streaming

Zhifei Xie and Changqiao Wu. Mini-omni: Language models can hear, talk while thinking in streaming. arXiv preprint arXiv:2408.16725, 2024

2024 arXiv
[224]

Mini-omni2: Towards open-source gpt-4o with vision, speech and duplex capabilities, 2024

Zhifei Xie and Changqiao Wu. Mini-omni2: Towards open-source gpt-4o with vision, speech and duplex capabilities, 2024

2024
[225]

Bigcodec: Pushing the limits of low-bitrate neural speech codec

Detai Xin, Xu Tan, Shinnosuke Takamichi, and Hiroshi Saruwatari. Bigcodec: Pushing the limits of low-bitrate neural speech codec. arXiv preprint arXiv:2409.05377, 2024

2024 arXiv
[226]

Mucodec: Ultra low-bitrate music codec

Yaoxun Xu, Hangting Chen, Jianwei Yu, Wei Tan, Rongzhi Gu, Shun Lei, Zhiwei Lin, and Zhiyong Wu. Mucodec: Ultra low-bitrate music codec. arXiv preprint arXiv:2409.13216, 2024

2024 arXiv
[227]

Spatialcodec: Neural spatial speech coding

Zhongweiyang Xu, Yong Xu, Vinay Kothapally, Heming Wang, Muqiao Yang, and Dong Yu. Spatialcodec: Neural spatial speech coding. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1131–1135. IEEE, 2024

2024
[228]

E- chat: Emotion-sensitive spoken dialogue system with large language models

Hongfei Xue, Yuhao Liang, Bingshen Mu, Shiliang Zhang, Qian Chen, and Lei Xie. E- chat: Emotion-sensitive spoken dialogue system with large language models. arXiv preprint arXiv:2401.00475, 2023

2023 arXiv
[229]

Qwen2 technical report

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024

2024 arXiv
[230]

Uniaudio 1.5: Large language model-driven audio codec is a few-shot audio task learner

Dongchao Yang, Haohan Guo, Yuanyuan Wang, Rongjie Huang, Xiang Li, Xu Tan, Xixin Wu, and Helen Meng. Uniaudio 1.5: Large language model-driven audio codec is a few-shot audio task learner. arXiv preprint arXiv:2406.10056, 2024

2024 arXiv
[231]

Hifi-codec: Group-residual vector quantization for high fidelity audio codec

Dongchao Yang, Songxiang Liu, Rongjie Huang, Jinchuan Tian, Chao Weng, and Yuexian Zou. Hifi-codec: Group-residual vector quantization for high fidelity audio codec. arXiv preprint arXiv:2305.02765, 2023

2023 arXiv
[232]

Instructtts: Modelling expressive tts in discrete latent space with natural language style prompt.IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024

Dongchao Yang, Songxiang Liu, Rongjie Huang, Chao Weng, and Helen Meng. Instructtts: Modelling expressive tts in discrete latent space with natural language style prompt.IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024

2024
[233]

Uniaudio: An audio foundation model toward universal audio generation

Dongchao Yang, Jinchuan Tian, Xu Tan, Rongjie Huang, Songxiang Liu, Xuankai Chang, Jiatong Shi, Sheng Zhao, Jiang Bian, Xixin Wu, et al. Uniaudio: An audio foundation model toward universal audio generation. arXiv preprint arXiv:2310.00704, 2023

arXiv
[234]

Generative de-quantization for neural speech codec via latent diffusion

Haici Yang, Inseon Jang, and Minje Kim. Generative de-quantization for neural speech codec via latent diffusion. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1251–1255. IEEE, 2024

2024
[235]

Air-bench: Benchmarking large audio-language models via generative comprehension

Qian Yang, Jin Xu, Wenrui Liu, Yunfei Chu, Ziyue Jiang, Xiaohuan Zhou, Yichong Leng, Yuanjun Lv, Zhou Zhao, Chang Zhou, et al. Air-bench: Benchmarking large audio-language models via generative comprehension. arXiv preprint arXiv:2402.07729, 2024. 56

2024 arXiv
[236]

Superb: Speech processing universal performance benchmark

Shu-wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y Lin, Andy T Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, et al. Superb: Speech processing universal performance benchmark. arXiv preprint arXiv:2105.01051, 2021

2021 arXiv
[237]

Codec does matter: Exploring the semantic shortcoming of codec for audio language model

Zhen Ye, Peiwen Sun, Jiahe Lei, Hongzhan Lin, Xu Tan, Zheqi Dai, Qiuqiang Kong, Jianyi Chen, Jiahao Pan, Qifeng Liu, Yike Guo, and Wei Xue. Codec does matter: Exploring the semantic shortcoming of codec for audio language model. arXiv preprint arXiv:2408.17175, 2024

2024 arXiv
[238]

Megabyte: Predicting million-byte sequences with multiscale transformers

Lili Yu, Dániel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zettlemoyer, and Mike Lewis. Megabyte: Predicting million-byte sequences with multiscale transformers. Advances in Neural Information Processing Systems, 36:78808–78823, 2023

2023
[239]

Soundstream: An end-to-end neural audio codec

Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495–507, 2021

2021
[240]

Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

1905 arXiv
[241]

Libritts: A corpus derived from librispeech for text-to-speech

Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. Libritts: A corpus derived from librispeech for text-to-speech. arXiv preprint arXiv:1904.02882, 2019

1904 arXiv
[242]

Wenetspeech: A 10000+ hours multi-domain mandarin corpus for speech recognition

Binbin Zhang, Hang Lv, Pengcheng Guo, Qijie Shao, Chao Yang, Lei Xie, Xin Xu, Hui Bu, Xiaoyu Chen, Chenchen Zeng, et al. Wenetspeech: A 10000+ hours multi-domain mandarin corpus for speech recognition. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and ...

2022
[243]

Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities

Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities. arXiv preprint arXiv:2305.11000, 2023

2023 arXiv
[244]

Speechalign: Aligning speech generation to human preferences

Dong Zhang, Zhaowei Li, Shimin Li, Xin Zhang, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. Speechalign: Aligning speech generation to human preferences. arXiv preprint arXiv:2404.05600, 2024

2024 arXiv
[245]

Speechgpt-gen: Scaling chain-of-information speech generation

Dong Zhang, Xin Zhang, Jun Zhan, Shimin Li, Yaqian Zhou, and Xipeng Qiu. Speechgpt-gen: Scaling chain-of-information speech generation. arXiv preprint arXiv:2401.13527, 2024

2024 arXiv
[246]

M4singer: A multi-style, multi-singer and musical score provided mandarin singing corpus

Lichao Zhang, Ruiqi Li, Shoutong Wang, Liqun Deng, Jinglin Liu, Yi Ren, Jinzheng He, Rongjie Huang, Jieming Zhu, Xiao Chen, et al. M4singer: A multi-style, multi-singer and musical score provided mandarin singing corpus. Advances in Neural Information Processing Systems, 35:69...

2022
[247]

Omniflatten: An end-to-end gpt model for seamless voice conversation

Qinglin Zhang, Luyao Cheng, Chong Deng, Qian Chen, Wen Wang, Siqi Zheng, Jiaqing Liu, Hai Yu, and Chaohong Tan. Omniflatten: An end-to-end gpt model for seamless voice conversation. arXiv preprint arXiv:2410.17799, 2024

2024 arXiv
[248]

Bertscore: Evaluating text generation with bert

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675, 2019

1904 arXiv
[249]

Intrinsicvoice: Empowering llms with intrinsic real-time voice interaction abilities

Xin Zhang, Xiang Lyu, Zhihao Du, Qian Chen, Dong Zhang, Hangrui Hu, Chaohong Tan, Tianyu Zhao, Yuxuan Wang, Bin Zhang, et al. Intrinsicvoice: Empowering llms with intrinsic real-time voice interaction abilities. arXiv preprint arXiv:2410.08035, 2024

arXiv
[250]

Speechtokenizer: Unified speech tokenizer for speech large language models

Xin Zhang, Dong Zhang, Shimin Li, Yaqian Zhou, and Xipeng Qiu. Speechtokenizer: Unified speech tokenizer for speech large language models. arXiv preprint arXiv:2308.16692, 2023

2023 arXiv
[251]

Speak foreign languages with your own voice: Cross-lingual neural codec language modeling

Ziqiang Zhang, Long Zhou, Chengyi Wang, Sanyuan Chen, Yu Wu, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Speak foreign languages with your own voice: Cross-lingual neural codec language modeling. arXiv preprint arXiv:2303.03926, 2023. 57

2023 arXiv
[252]

Comparison of different implementations of mfcc

Fang Zheng, Guoliang Zhang, and Zhanjiang Song. Comparison of different implementations of mfcc. Journal of Computer science and Technology, 16:582–589, 2001

2001
[253]

Judging llm-as-a-judge with mt-bench and chatbot arena

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595– 46623, 2023

2023
[254]

Fuvc: A flexible codec for underwater video transmission

Yannan Zheng, Jiawei Luo, Weiling Chen, Zuoyong Li, and Tiesong Zhao. Fuvc: A flexible codec for underwater video transmission. IEEE Transactions on Geoscience and Remote Sensing, 2024

2024
[255]

Supercodec: A neural speech codec with selective back-projection network

Youqiang Zheng, Weiping Tu, Li Xiao, and Xinmeng Xu. Supercodec: A neural speech codec with selective back-projection network. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 566–570. IEEE, 2024

2024
[256]

Agieval: A human-centric benchmark for evaluating foundation models

Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models. arXiv preprint arXiv:2304.06364, 2023

2023 arXiv
[257]

Transfusion: Predict the next token and diffuse images with one multi-modal model

Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model. arXiv preprint arXiv:2408.11039, 2024

2024 arXiv
[258]

Addressing representation collapse in vector quantized models with one linear layer

Yongxin Zhu, Bocheng Li, Yifei Xin, and Linli Xu. Addressing representation collapse in vector quantized models with one linear layer. arXiv preprint arXiv:2411.02038, 2024. A Resources about Music and Sound Datasets This section lists commonly used music and sound datasets. T...

2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [213]

Freeze-omni: A smart and low latency speech-to-speech dialogue model with frozen llm, 2024

Xiong Wang, Yangze Li, Chaoyou Fu, Lei Xie, Ke Li, Xing Sun, and Long Ma. Freeze-omni: A smart and low latency speech-to-speech dialogue model with frozen llm, 2024

2024

[2] [214]

Freeze-omni: A smart and low latency speech-to-speech dialogue model with frozen llm

Xiong Wang, Yangze Li, Chaoyou Fu, Lei Xie, Ke Li, Xing Sun, and Long Ma. Freeze-omni: A smart and low latency speech-to-speech dialogue model with frozen llm. arXiv preprint arXiv:2411.00774, 2024

arXiv 2024

[3] [1]

Gpt-4 technical report

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

arXiv 2023

[4] [2]

Musiclm: Generating music from text

Andrea Agostinelli, Timo I Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al. Musiclm: Generating music from text. arXiv preprint arXiv:2301.11325, 2023

arXiv 2023

[5] [3]

Hilcodec: High fidelity and lightweight neural audio codec

Sunghwan Ahn, Beom Jun Woo, Min Hyun Han, Chanyeong Moon, and Nam Soo Kim. Hilcodec: High fidelity and lightweight neural audio codec. arXiv preprint arXiv:2405.04752, 2024

arXiv 2024

[6] [4]

Apcodec: A neural audio codec with parallel amplitude and phase spectrum encoding and decoding

Yang Ai, Xiao-Hang Jiang, Ye-Xin Lu, Hui-Peng Du, and Zhen-Hua Ling. Apcodec: A neural audio codec with parallel amplitude and phase spectrum encoding and decoding. arXiv preprint arXiv:2402.10533, 2024

arXiv 2024

[7] [5]

Seed-tts: A family of high-quality versatile speech generation models

Philip Anastassiou, Jiawei Chen, Jitong Chen, Yuanzhe Chen, Zhuo Chen, Ziyi Chen, Jian Cong, Lelai Deng, Chuang Ding, Lu Gao, et al. Seed-tts: A family of high-quality versatile speech generation models. arXiv preprint arXiv:2406.02430, 2024

arXiv 2024

[8] [6]

Palm 2 technical report

Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023

arXiv 2023