Pith. sign in

REVIEW 26 cited by

Mini-Omni2: Towards Open-source GPT-4o with Vision, Speech and Duplex Capabilities

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.11190 v3 pith:24VY7DOQ submitted 2024-10-15 eess.AS cs.AIcs.CVcs.LGcs.SD

Zhifei Xie , Changqiao Wu This is my paper · ORCID
classification eess.AScs.AIcs.CVcs.LGcs.SD
keywords gpt-4omini-omni2modalitiesmodeltraininginteractionmulti-modalvisual
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

GPT-4o, an all-encompassing model, represents a milestone in the development of large multi-modal language models. It can understand visual, auditory, and textual modalities, directly output audio, and support flexible duplex interaction. Models from the open-source community often achieve some functionalities of GPT-4o, such as visual understanding and voice chat. Nevertheless, training a unified model that incorporates all modalities is challenging due to the complexities of multi-modal data, intricate model architectures, and training processes. In this paper, we introduce Mini-Omni2, a visual-audio assistant capable of providing real-time, end-to-end voice responses to visoin and audio queries. By integrating pretrained visual and auditory encoders, Mini-Omni2 maintains performance in individual modalities. We propose a three-stage training process to align modalities, allowing the language model to handle multi-modal inputs and outputs after training on a limited dataset. For interaction, we introduce a command-based interruption mechanism, enabling more flexible interaction with users. To the best of our knowledge, Mini-Omni2 is one of the closest reproductions of GPT-4o, which have similar form of functionality, and we hope it can offer valuable insights for subsequent research.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 26 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. VideoFDB: Evaluating Full-Duplex Vision-Speech Capabilities in Conversational Agents

    cs.CV 2026-05 unverdicted novelty 8.0 of 10

    VideoFDB is a new benchmark and LM-as-judge framework for evaluating full-duplex audio-visual-to-audio-visual conversational agents on nonverbal dynamics from real video calls.

  2. SocialOmni: Benchmarking Audio-Visual Social Interactivity in Omni Models

    cs.AI 2026-03 conditional novelty 6.5 of 10

    SocialOmni jointly evaluates speaker ID, turn-entry timing, and interruption phrasing on 12 OLMs and finds perception accuracy decouples from socially appropriate generation.

  3. Cocktail-Talker: Multi-Speaker Dialog Modeling in Noisy Social Environments with Turn Action GRPO

    cs.SD 2026-07 conditional novelty 6.0 of 10

    Cocktail-Talker uses three action tokens and GRPO to make a speech LLM decide whether to respond, keep listening, or ignore audio in noisy multi-speaker conversations.

  4. Let Me Look at You: Advanced Facial Expression Modeling for Conversational Speech Synthesis

    cs.HC 2026-07 conditional novelty 6.0 of 10

    AU-supervised single-token face encoding plus dual visual–speech DPO on a large real-conversation dataset improves empathetic conversational TTS over text/speech-only and prior visual CSS systems.

  5. OmniScope: Modality-Decoupled Token Compression for Omnimodal Large Language Models

    cs.CV 2026-07 conditional novelty 6.0 of 10

    A training-free, query-anchored, modality-decoupled token compression method outperforms unidirectional audio/video compression baselines on Qwen2.5-Omni while cutting prefill cost.

  6. Hierarchical Acoustic-Semantic Modeling: Modality Separation and Semantic Coherence for Full-Duplex SLMs

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Lychee-FD resolves modality interference in full-duplex spoken language models by separating acoustic and semantic parameters in deep layers and adding a dense semantic alignment channel, achieving state-of-the-art pe...

  7. Entropy-based Coarse and Compressed Semantic Speech Representation Learning

    cs.CL 2025-08 conditional novelty 6.0 of 10

    Predictive entropy from a token-level speech language model finds merge boundaries, producing compressed semantic tokens that keep ASR and translation accuracy at 15 Hz while lowering latency.

  8. LLaSO: A Foundational Framework for Reproducible Research in Large Language and Speech Model

    cs.CL 2025-08 unverdicted novelty 6.0 of 10

    LLaSO releases a 3.8B speech-language model, 25.5M training instances, and an evaluation benchmark, claiming a normalized score of 0.72.

  9. AURA: Agent for Understanding, Reasoning, and Automated Tool Use in Voice-Driven Tasks

    cs.AI 2025-06 conditional novelty 6.0 of 10

    A cascaded open-weight speech agent using ReAct reasoning and external tools reaches 92.75% on VoiceBench OpenBookQA and 90% success on 30 multi-turn voice tasks.

  10. OpusLM: A Family of Open Unified Speech Language Models

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A new open family of speech-language models trained on public data matches or beats prior systems across ASR, TTS, and text benchmarks.

  11. AV-Reasoner: Improving and Benchmarking Clue-Grounded Audio-Visual Counting for MLLMs

    cs.CV 2025-06 reject novelty 6.0 of 10

    A clue-grounded audio-visual counting benchmark over 497 long videos and an RL-trained counting model, whose headline result is undermined by training on the DVD-Counting evaluation benchmark.

  12. SOVA-Bench: Benchmarking the Speech Conversation Ability for LLM-based Voice Assistant

    cs.SD 2025-06 conditional novelty 6.0 of 10

    SOVA-Bench is a new evaluation framework for speech LLMs covering knowledge, recognition, linguistic and paralinguistic understanding, and semantic and acoustic generation quality.

  13. SALM-Duplex: Efficient and Direct Duplex Modeling for Speech-to-Speech Language Model

    cs.CL 2025-05 conditional novelty 6.0 of 10

    A speech-to-speech language model uses channel fusion of a streaming encoder and codec tokens to handle barge-in and turn-taking without speech pretraining, showing improved metrics over Moshi at 0.6 kbps.

  14. OmniFocus: Query-Guided Modality-Balanced Token Compression for Omni-Modal Large Language Models

    cs.LG 2026-07 conditional novelty 5.5 of 10

    Query-guided, modality-symmetric token compression preserves audio-visual evidence better than audio-guided budgets at 25–35% retention on Qwen2.5-Omni.

  15. ALAS: An Automatic Latent Alignment Score for Audio Language Models

    cs.CL 2025-05 conditional novelty 5.0 of 10

    ALAS is a reference-based score for audio-text alignment in speech LLMs, computed from frozen hidden states and a Whisper-derived alignment path, with no training or fitted classifier.

  16. LiSTEN: Learning Soft Token Embeddings for Neural Audio LLMs

    cs.AI 2025-05 conditional novelty 5.0 of 10

    LiSTEN shows that dynamically selecting a few learnable prompt tokens from a shared pool can replace LoRA fine-tuning for audio-language models, matching or beating it with less training data.

  17. Ola: Pushing the Frontiers of Omni-Modal Language Model

    cs.CV 2025-02 conditional novelty 5.0 of 10

    Ola, a 7B omni-modal language model, achieves competitive image, video, and audio understanding with progressive modality alignment, though it does not beat all specialized models on every benchmark.

  18. MinMo: A Multimodal Large Language Model for Seamless Voice Interaction

    cs.CL 2025-01 conditional novelty 5.0 of 10

    A single aligned multimodal model, MinMo, achieves strong or state-of-the-art results on speech recognition, translation, emotion recognition, voice generation, and full-duplex dialogue while preserving the underlying...

  19. SLAM-Omni: Timbre-Controllable Voice Interaction System with Single-Stage Training

    eess.AS 2024-12 conditional novelty 5.0 of 10

    A 0.5B spoken dialogue model trained end-to-end in one stage, with grouped semantic tokens for faster generation and text-only history for multi-turn dialogue.

  20. Advancing Speech Language Models by Scaling Supervised Fine-Tuning with Over 60,000 Hours of Synthetic Speech Dialogue Data

    cs.CL 2024-12 conditional novelty 5.0 of 10

    A 60,000-hour bilingual synthetic speech dialogue dataset and a trained speech language model demonstrate strong speech interaction performance.

  21. OpenGlass: A Sensing-Computing Split Architecture for Local MLLM-Driven Real-Time Visual Assistance

    cs.CV 2026-07 conditional novelty 4.0 of 10

    A sensing-computing split (ESP32 glasses + nearby consumer GPU) delivers ~993 ms median local MLLM visual assistance with safety abstention and open artifacts.

  22. Speech Discrete Tokens or Continuous Features? A Comparative Analysis for Spoken Language Understanding in SpeechLLMs

    cs.CL 2025-08 unverdicted novelty 4.0 of 10

    Under matched settings, continuous SSL speech features generally outperform discrete tokens on six spoken language understanding tasks in SpeechLLMs.

  23. HumanOmni: A Large Vision-Speech Language Model for Human-Centric Video Understanding

    cs.CV 2025-01 reject novelty 4.0 of 10

    A human-centric vision-speech language model with three instruction-weighted visual branches and audio input reports strong emotion, facial expression, and action results, but its evaluation is under-specified and not...

  24. Real-Time Textless Dialogue Generation

    cs.CL 2025-01 conditional novelty 4.0 of 10

    A streaming, textless dialogue model predicts turn-taking actions every 160 ms and generates speech units, improving naturalness over cascaded systems at the cost of lower semantic coherence.

  25. AlignFormer: Modality Matching Can Achieve Better Zero-shot Instruction-Following Speech-LLM

    eess.AS 2024-12 conditional novelty 4.0 of 10

    AlignFormer, a CTC-guided dynamic-window adapter, lets a frozen LLM trained on ASR data alone achieve high instruction-following rates on zero-shot speech translation and question answering.

  26. Visual Error Patterns in Multi-Modal AI: A Statistical Approach

    cs.LG 2024-11 conditional novelty 3.0 of 10

    A gradient boosting model predicts GPT-4o visual classification errors on 75 geometric stimuli with AUC 0.85, with 3D and Pentagon features dominating.

Pith tools