Pith. sign in

REVIEW 11 cited by

Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Experts

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2405.11273 v1 pith:HSEB325X submitted 2024-05-18 cs.AI cs.CLcs.CVcs.MM

Yunxin Li , Shenyuan Jiang , Baotian Hu , Longyue Wang , Wanqi Zhong , Wenhan Luo , Lin Ma , Min Zhang This is my paper · ORCID
classification cs.AIcs.CLcs.CVcs.MM
keywords multimodaldataexpertsuni-moearchitecturecross-modalitymodelstraining
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Recent advancements in Multimodal Large Language Models (MLLMs) underscore the significance of scalable models and data to boost performance, yet this often incurs substantial computational costs. Although the Mixture of Experts (MoE) architecture has been employed to efficiently scale large language and image-text models, these efforts typically involve fewer experts and limited modalities. To address this, our work presents the pioneering attempt to develop a unified MLLM with the MoE architecture, named Uni-MoE that can handle a wide array of modalities. Specifically, it features modality-specific encoders with connectors for a unified multimodal representation. We also implement a sparse MoE architecture within the LLMs to enable efficient training and inference through modality-level data parallelism and expert-level model parallelism. To enhance the multi-expert collaboration and generalization, we present a progressive training strategy: 1) Cross-modality alignment using various connectors with different cross-modality data, 2) Training modality-specific experts with cross-modality instruction data to activate experts' preferences, and 3) Tuning the Uni-MoE framework utilizing Low-Rank Adaptation (LoRA) on mixed multimodal instruction data. We evaluate the instruction-tuned Uni-MoE on a comprehensive set of multimodal datasets. The extensive experimental results demonstrate Uni-MoE's principal advantage of significantly reducing performance bias in handling mixed multimodal datasets, alongside improved multi-expert collaboration and generalization. Our findings highlight the substantial potential of MoE frameworks in advancing MLLMs and the code is available at https://github.com/HITsz-TMG/UMOE-Scaling-Unified-Multimodal-LLMs.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. M3LLM: Model Context Protocol-aided Mixture of Vision Experts For Multimodal LLMs in Networks

    cs.NI 2025-08 reject novelty 6.0 of 10

    M3LLM routes each multimodal query to the semantically most suitable, wirelessly reachable vision expert using protocol-aided retrieval and a decoupled reinforcement learning agent.

  2. Chain-of-Experts: Unlocking the Communication Power of Mixture-of-Experts Models

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Chain-of-Experts replaces one parallel MoE routing step with several sequential expert steps inside a layer, reporting lower loss and memory use in small-scale experiments.

  3. MoHAVE: Mixture of Hierarchical Audio-Visual Experts for Robust Speech Recognition

    eess.AS 2025-02 conditional novelty 6.0 of 10

    A decoder with modal-specialized expert groups and a learned two-level router improves noisy audio-visual speech recognition on LRS3 and MuAViC, using roughly half the active parameter count of the dense baseline.

  4. MoVE-KD: Knowledge Distillation for VLMs with Mixture of Visual Encoders

    cs.CV 2025-01 conditional novelty 6.0 of 10

    MoVE-KD distills multiple visual encoders into one student encoder using mixture-of-LoRA-experts and CLS-attention-guided distillation, improving LLaVA-family VLMs at single-encoder cost.

  5. DeMo: Decoupled Feature-Based Mixture of Experts for Multi-Modal Object Re-Identification

    cs.CV 2024-12 conditional novelty 6.0 of 10

    DeMo improves multi-modal object re-identification by decoupling RGB, NIR, and TIR features into seven attention-derived streams and weighting them with an attention-triggered mixture of experts.

  6. Neural Inhibition Improves Dynamic Routing and Mixture of Experts

    cs.LG 2025-07 conditional novelty 5.0 of 10

    Neural inhibition gating on MoE router inputs improves a synthetic digit/squares benchmark by about four points over plain MoE, but the language-model evidence is unreliable.

  7. MergeME: Model Merging Techniques for Homogeneous and Heterogeneous MoEs

    cs.CL 2025-02 conditional novelty 5.0 of 10

    Merging specialized LLMs into a MoE can be improved by replacing averaging with Dare/Ties merging and by using perplexity-based routing, while heterogeneous experts can be merged with projectors and a sequence-level router.

  8. SynerGen-VL: Towards Synergistic Image Understanding and Generation with Vision Experts and Token Folding

    cs.CV 2024-12 conditional novelty 5.0 of 10

    SynerGen-VL introduces token folding and vision expert FFNs to train a 2.4B encoder-free MLLM that matches larger unified models like Emu3 on multiple image understanding and generation benchmarks.

  9. Decoding Visual Neural Representations by Multimodal with Dynamic Balancing

    cs.CV 2025-09 conditional novelty 4.0 of 10

    A multimodal EEG-image-text contrastive framework with dynamic gradient balancing and stochastic noise improves zero-shot object recognition from EEG on ThingsEEG, raising top-1 accuracy from 13.8% to 15.8%.

  10. 3D-MoE: A Mixture-of-Experts Multi-modal LLM for 3D Vision and Pose Diffusion via Rectified Flow

    cs.CL 2025-01 conditional novelty 4.0 of 10

    A 1B-parameter MoE multi-modal LLM for 3D tasks, paired with a rectified-flow pose diffusion head, reports better scores than 7B baselines on 3D QA and manipulation planning.

  11. Align Anything: Training All-Modality Models to Follow Instructions with Language Feedback

    cs.AI 2024-12 conditional novelty 4.0 of 10

    The paper proposes learning from language feedback to synthesize multimodal preference pairs, but the evidence is weakened by an undefined improvement metric and small, unvalidated effect sizes.

Pith tools