REVIEW 53 cited by
Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Recent advances in language models have achieved significant progress. GPT-4o, as a new milestone, has enabled real-time conversations with humans, demonstrating near-human natural fluency. Such human-computer interaction necessitates models with the capability to perform reasoning directly with the audio modality and generate output in streaming. However, this remains beyond the reach of current academic models, as they typically depend on extra TTS systems for speech synthesis, resulting in undesirable latency. This paper introduces the Mini-Omni, an audio-based end-to-end conversational model, capable of real-time speech interaction. To achieve this capability, we propose a text-instructed speech generation method, along with batch-parallel strategies during inference to further boost the performance. Our method also helps to retain the original model's language capabilities with minimal degradation, enabling other works to establish real-time interaction capabilities. We call this training method "Any Model Can Talk". We also introduce the VoiceAssistant-400K dataset to fine-tune models optimized for speech output. To our best knowledge, Mini-Omni is the first fully end-to-end, open-source model for real-time speech interaction, offering valuable potential for future research.
Forward citations
Cited by 53 Pith papers
-
ProsodyLM: Uncovering the Emerging Prosody Processing Capabilities in Speech Language Models
A speech language model trained only on audiobooks with explicit word-level prosody tokens displays emerging abilities in prosody-controlled generation, emphasis and emotion understanding, and prosodic consistency acr...
-
Unlocking Temporal Flexibility: Neural Speech Codec with Variable Frame Rate
A neural speech codec that dynamically varies frame rate per segment using waveform entropy achieves competitive or better reconstruction quality at lower average frame rates than constant frame rate baselines.
-
Scaling Speech-Text Pre-training with Synthetic Interleaved Data
Synthetic interleaved speech-text data generated from text corpora lets a 9B speech LM reach SOTA spoken QA (31% vs 13% previous average SOTA).
-
Metronome: Bound the Cache, Keep the Beat for Real-Time Interaction Model Serving
Bounding per-session resident KV eliminates the silent metastable latency cliff in real-time interaction serving and turns frame latency into a usable admission signal.
-
Towards Expert-level Medical AI for Real-time Video Consultations
A video-based medical AI system matched or outperformed primary care physicians in a randomized 100-scenario simulated telehealth study, while patients still preferred human doctors for rapport.
-
EmoS: A Theory-Grounded Framework for Evaluating and Aligning Emotional Intelligence in Spoken Language Models
A new four-branch benchmark, EmoDialogue dataset, and GRPO-trained evaluator (EmoS) claim near-human emotional intelligence scoring for spoken language models.
-
Cocktail-Talker: Multi-Speaker Dialog Modeling in Noisy Social Environments with Turn Action GRPO
Cocktail-Talker uses three action tokens and GRPO to make a speech LLM decide whether to respond, keep listening, or ignore audio in noisy multi-speaker conversations.
-
Efficient Chain-of-Modality Reasoning via Progressive Compression for Spoken Language Models
Spoken math models that emit a 40%-compressed reasoning trace between question and answer beat full-reasoning baselines by ~3 accuracy points while using roughly one third of the text tokens.
-
Hierarchical Acoustic-Semantic Modeling: Modality Separation and Semantic Coherence for Full-Duplex SLMs
Lychee-FD resolves modality interference in full-duplex spoken language models by separating acoustic and semantic parameters in deep layers and adding a dense semantic alignment channel, achieving state-of-the-art pe...
-
SPEARBench: A Benchmark for Naturalness Evaluation in Streaming Speech-to-Speech Language Models
An open-source benchmark for speech-to-speech models shows that current systems produce intelligible audio but diverge from human conversational behavior in latency, dialect consistency, emotional entrainment, and prosody.
-
TurnNat: Automatic Evaluation of Turn-Taking Naturalness in Dyadic Spoken Dialogue
TurnNat introduces a likelihood-based automatic evaluation method for turn-taking naturalness in dyadic spoken dialogues using a causal prediction model and a human-validated perturbation benchmark.
-
A Synonymous Variational Perspective on the Rate-Distortion-Perception Tradeoff
Synset-based reconstruction and synonymous variational inference are claimed to derive the distributional divergence in RDP and unify it with classical rate-distortion theory.
-
Omni-Diffusion: Unified Multimodal Understanding and Generation with Masked Discrete Diffusion
A unified mask-based discrete diffusion model jointly models text, speech, and image tokens and matches or exceeds several any-to-any and specialist multimodal baselines.
-
AV-EMO-Reasoning: Benchmarking Emotional Reasoning Capabilities in Omni-modal LLMS with Audio-visual Cues
Current omni-modal LLMs underperform on audio-visual emotional reasoning, and automatic scores diverge from human perceptual judgments; AV-EMO-Reasoning provides a benchmark to measure this.
-
UniVoice: Unifying Autoregressive ASR and Flow-Matching based TTS with Large Language Models
A single LLM can do ASR and zero-shot TTS on continuous speech features by switching between causal and bidirectional attention, reaching competitive but not state-of-the-art results.
-
KAME: Tandem Architecture for Enhancing Knowledge in Real-Time Speech-to-Speech Conversational AI
By injecting a back-end LLM's text responses into a streaming speech-to-speech model as oracle tokens, KAME improves MT-Bench answer scores from 2.05 to 6.43 at unchanged median latency.
-
Entropy-based Coarse and Compressed Semantic Speech Representation Learning
Predictive entropy from a token-level speech language model finds merge boundaries, producing compressed semantic tokens that keep ASR and translation accuracy at 15 Hz while lowering latency.
-
LLaSO: A Foundational Framework for Reproducible Research in Large Language and Speech Model
LLaSO releases a 3.8B speech-language model, 25.5M training instances, and an evaluation benchmark, claiming a normalized score of 0.72.
-
FD-Bench: A Full-Duplex Benchmarking Pipeline Designed for Full Duplex Spoken Dialogue Systems
FD-Bench is a new LLM/TTS/ASR-based benchmark for full-duplex spoken dialogue systems, and applying it to Moshi, Freeze-omni, and VITA-1.5 shows all three struggle with frequent interruptions and noise.
-
AURA: Agent for Understanding, Reasoning, and Automated Tool Use in Voice-Driven Tasks
A cascaded open-weight speech agent using ReAct reasoning and external tools reaches 92.75% on VoiceBench OpenBookQA and 90% success on 30 multi-turn voice tasks.
-
SOVA-Bench: Benchmarking the Speech Conversation Ability for LLM-based Voice Assistant
SOVA-Bench is a new evaluation framework for speech LLMs covering knowledge, recognition, linguistic and paralinguistic understanding, and semantic and acoustic generation quality.
-
StarVC: A Unified Auto-Regressive Framework for Joint Text and Speech Generation in Voice Conversion
StarVC is an autoregressive voice conversion model that generates text tokens before acoustic tokens, improving linguistic fidelity while retaining speaker similarity.
-
NTPP: Generative Speech Language Modeling for Dual-Channel Spoken Dialogue via Next-Token-Pair Prediction
NTPP models dual-channel spoken dialogue by predicting both speakers' next speech tokens as a pair, achieving speaker-independent full-duplex generation in a decoder-only transformer.
-
Zero-Shot Streaming Text to Speech Synthesis with Transducer and Auto-Regressive Modeling
SMLLE generates speech frame-by-frame using a Transducer for streaming semantic tokens plus a fully autoregressive mel-spectrogram model, reaching quality close to sentence-level zero-shot TTS.
-
Audio Jailbreak: An Open Comprehensive Benchmark for Jailbreaking Large Audio-Language Models
AJailBench is an open benchmark showing that large audio-language models can be jailbroken through TTS-converted text attacks and through subtle acoustic perturbations that preserve speech semantics.
-
SALMONN-omni: A Standalone Speech LLM without Codec Injection for Full-duplex Conversation
SALMONN-omni is a standalone full-duplex speech LLM that interleaves continuous speech and text embeddings in one model and uses special thinking tokens to learn turn-taking and barge-in behavior.
-
SimulS2S-LLM: Unlocking Simultaneous Inference of Speech LLMs for Speech-to-Speech Translation
An offline-trained speech LLM with boundary-aware CIF speech prompts and test-time wait-k decoding achieves better quality-latency trade-offs in simultaneous speech-to-speech translation than StreamSpeech on CVSS-C.
-
VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Models
On VisuLogic's 1,000 vision-centric puzzles, the best multimodal models reach 28.1% accuracy versus a 24.9% random baseline and 51.4% human accuracy, and an RL baseline lifts accuracy by up to 5.6 points.
-
Neural Codec Source Tracing: Toward Comprehensive Attribution in Open-Set Condition
A new open-set neural codec source tracing benchmark and dataset shows strong in-distribution classification and OOD detection, but poor generalization to unseen real audio.
-
SocialMind: LLM-based Proactive AR Social Assistive System with Human-like Perception for In-situ Live Interactions
SocialMind provides real-time, proactive social suggestions on AR glasses by combining multimodal sensing, persona memory, and LLM reasoning.
-
OmniFocus: Query-Guided Modality-Balanced Token Compression for Omni-Modal Large Language Models
Query-guided, modality-symmetric token compression preserves audio-visual evidence better than audio-guided budgets at 25–35% retention on Qwen2.5-Omni.
-
X$^3$-OPD: Distilling Reasoning into Large Audio-Language Models via On-Policy Alignment
X3-OPD improves audio-grounded reasoning by training the audio student on its own rollouts with token-level teacher feedback, using a three-tier paired text-audio corpus.
-
Sharp spectral estimates for free boundary problems arising in plasma physics
For a constrained superlinear free-boundary plasma model, the non-local first eigenvalue σ₁ is always positive on balls in every dimension N≥2, despite lacking a general Faber–Krahn property.
-
FireRedChat: A Pluggable, Full-Duplex Voice Interaction System with Cascaded and Semi-Cascaded Implementations
A full-duplex voice system with streaming personalized VAD and semantic end-of-turn detection reports fewer false barge-ins and latencies near commercial benchmarks.
-
ChipChat: Low-Latency Cascaded Conversational Agent in MLX
ChipChat is a fully on-device cascaded voice agent that reports about 920 ms total latency using streaming ASR, a state-action LLM, streaming TTS, and a vocoder.
-
Stream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Model
Stream-Omni uses CTC-based layer-dimension mapping to align speech with text, achieving vision, speech, and text interaction in one 8B model trained on 23,000 hours of speech.
-
RoboEgo System Card: An Omnimodal Model with Native Full Duplexity
RoboEgo combines a full-duplex audio/text backbone with vision and action heads to achieve an 80 ms theoretical response granularity, reporting competitive quality and better responsiveness than Qwen2.5-Omni in a smal...
-
Enhancing Generalization of Speech Large Language Models with Multi-Task Behavior Imitation and Speech-Text Interleaving
Multi-task behavior imitation with speech-text interleaving improves speech LLM generalization on prompts and zero-shot tasks using only paired speech and transcripts.
-
EASY: Emotion-aware Speaker Anonymization via Factorized Distillation
EASY separates speaker identity, linguistic content, and emotion through sequential factorized distillation, and reports better privacy and emotion preservation than prior VoicePrivacy 2024 systems.
-
Granite-speech: open-source speech-aware LLMs with strong English ASR capabilities
Granite-speech-3.3-2b and Granite-speech-3.3-8b achieve competitive English ASR word error rates, with the 8B model beating several larger proprietary models on multiple public benchmarks while remaining fully open-source.
-
LLaMA-Omni2: LLM-based Real-time Spoken Chatbot with Autoregressive Streaming Speech Synthesis
A series of Qwen2.5-based speech chatbots (0.5B to 14B) that stream speech output with a CosyVoice2-style autoregressive decoder and outperform prior SpeechLMs using only 200K synthetic multi-turn dialogues.
-
A Preliminary Exploration with GPT-4o Voice Mode
GPT-4o voice mode is evaluated on 180 Dynamic-SUPERB tasks plus MMAU and CMM, showing strong audio understanding and low hallucination, but unstable refusal behavior and weak duration and instrument skills.
-
The ICME 2025 Audio Encoder Capability Challenge
The paper is a call for participation in the ICME 2025 Audio Encoder Capability Challenge, describing the evaluation protocol, datasets, and scoring rules.
-
SLAM-Omni: Timbre-Controllable Voice Interaction System with Single-Stage Training
A 0.5B spoken dialogue model trained end-to-end in one stage, with grouped semantic tokens for faster generation and text-only history for multi-turn dialogue.
-
InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactions
The authors integrate streaming perception, compressed long-term memory, and a reasoning model into one open-source system, reporting SOTA open-source results on several video and audio benchmarks.
-
SALMONN-omni: A Codec-free LLM for Full-duplex Speech Understanding and Generation
SALMONN-omni proposes a codec-free, full-duplex speech LLM with state-transition 'thinking' tokens, but provides only qualitative case studies as evidence.
-
ILT-Iterative LoRA Training through Focus-Feedback-Fix for Multilingual Speech Recognition
A three-stage iterative LoRA training recipe (Focus, Feed Back, Fix) is applied to Whisper-large-v3 and Qwen2-Audio, reporting WER reductions on a multilingual ASR benchmark, with the gains attributed to the iterative...
-
Analyzing Mitigation Strategies for Catastrophic Forgetting in End-to-End Training of Spoken Language Models
In a three-stage end-to-end spoken language model, experience replay (mixing old data into later training) was the most effective mitigation against catastrophic forgetting, greatly outperforming model merging and LoR...
-
Overview of the Amphion Toolkit (v0.2)
Amphion v0.2 is an open-source toolkit for audio, music, and speech generation, adding a 101K-hour multilingual dataset, processing pipelines, and pretrained models.
-
SilVar: Speech Driven Multimodal Model for Reasoning Visual Question Answering and Object Localization
SilVar fuses speech, image, and text via CLIP, Whisper, and LLaMA to perform reasoning visual question answering and object localization, but its claimed state-of-the-art results are not supported by its own tables.
-
YingSound: Video-Guided Sound Effects Generation with Multi-modal Chain-of-Thought Controls
A video-guided sound effects model with a learnable audio-visual aggregator and multi-modal chain-of-thought module reports strong VGGSound benchmark scores, but its few-shot claim rests on three qualitative samples.
-
Continuous Speech Tokens Makes LLMs Robust Multi-Modality Learners
A speech-to-speech dialogue model that predicts mel-spectrograms with flow matching, jointly trained with discrete text tokens, achieves lower WER than a discrete speech token baseline.
-
WavChat: A Survey of Spoken Dialogue Models
WavChat categorizes spoken dialogue models into cascaded and end-to-end paradigms and surveys speech representations, training strategies, streaming, duplex interaction, datasets, and evaluation benchmarks.
Discussion (0). Continue with ORCID to comment.