Pith. sign in

REVIEW 32 cited by

A Survey on Mixture of Experts in Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2407.06204 v3 pith:NFSBEUEJ submitted 2024-06-26 cs.LG cs.CL

Weilin Cai , Juyong Jiang , Fan Wang , Jing Tang , Sunghun Kim , Jiayi Huang This is my paper
classification cs.LGcs.CL
keywords modelslanguagellmsdiverseexpertslargemixturemodel
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large language models (LLMs) have garnered unprecedented advancements across diverse fields, ranging from natural language processing to computer vision and beyond. The prowess of LLMs is underpinned by their substantial model size, extensive and diverse datasets, and the vast computational power harnessed during training, all of which contribute to the emergent abilities of LLMs (e.g., in-context learning) that are not present in small models. Within this context, the mixture of experts (MoE) has emerged as an effective method for substantially scaling up model capacity with minimal computation overhead, gaining significant attention from academia and industry. Despite its growing prevalence, there lacks a systematic and comprehensive review of the literature on MoE. This survey seeks to bridge that gap, serving as an essential resource for researchers delving into the intricacies of MoE. We first briefly introduce the structure of the MoE layer, followed by proposing a new taxonomy of MoE. Next, we overview the core designs for various MoE models including both algorithmic and systemic aspects, alongside collections of available open-source implementations, hyperparameter configurations and empirical evaluations. Furthermore, we delineate the multifaceted applications of MoE in practice, and outline some potential directions for future research. To facilitate ongoing updates and the sharing of cutting-edge advances in MoE research, we have established a resource repository at https://github.com/withinmiaov/A-Survey-on-Mixture-of-Experts-in-LLMs.

Discussion (0). Sign in to comment.

Forward citations

Cited by 32 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Optimal Mixture-of-Experts Model Averaging for Conditional Generative Models

    stat.ML 2026-07 accept novelty 6.5 of 10

    Sample-based MMD model averaging of conditional generators is asymptotically optimal, and input-adaptive MoEMA weights improve over fixed averaging and single models across modalities.

  2. SmartMage: Dynamic Modality Orchestration for 3D Scene Understanding

    cs.CV 2026-08 conditional novelty 6.0 of 10

    A multimodal language model that dynamically routes queries to the most relevant scene modalities and modality-specialized experts, achieving state-of-the-art results on five 3D benchmarks.

  3. Accelerating Mixture-of-Expert Inference with Adaptive Expert Split Mechanism

    cs.LG 2025-09 conditional novelty 6.0 of 10

    MoEpic splits each MoE expert into a cached top segment and a prefetched bottom segment, raising cache hit rates and hiding loading latency to accelerate offloaded MoE inference.

  4. MoFRR: Mixture of Diffusion Models for Face Retouching Restoration

    cs.CV 2025-07 conditional novelty 6.0 of 10

    MoFRR, a mixture of diffusion models with one specialized expert per retouching type plus a shared expert, restores original faces from retouched images and outperforms general restoration baselines on the RetouchingF...

  5. Improving Data and Parameter Efficiency of Neural Language Models Using Representation Analysis

    cs.CL 2025-07 conditional novelty 6.0 of 10

    Representation smoothness can be used to regularize training, stop early without validation labels, and guide active learning combined with parameter-efficient fine-tuning, reducing data and compute.

  6. Wavelet Policy: Lifting Scheme for Policy Learning in Long-Horizon Tasks

    cs.RO 2025-07 conditional novelty 6.0 of 10

    A learnable wavelet lifting network for policy learning improves long-horizon task performance across five simulation benchmarks.

  7. Scaling Intelligence: Designing Data Centers for Next-Gen Language Models

    cs.AR 2025-06 conditional novelty 6.0 of 10

    A simulation study argues that flat optical networks with uniform bandwidth improve LLM training throughput and reduce tuning sensitivity compared to two-tier networks.

  8. Predictable Scale: Part II, Farseer: A Refined Scaling Law in Large Language Models

    cs.LG 2025-06 conditional novelty 6.0 of 10

    A newly fitted scaling law with model-size-dependent data exponents predicts LLM loss more accurately than Chinchilla, including at a held-out 25.1B model.

  9. NoLoCo: No-all-reduce Low Communication Training Method for Large Models

    cs.LG 2025-06 conditional novelty 6.0 of 10

    NoLoCo trains large language models without any all-to-all synchronization by using pairwise weight averaging and random pipeline routing, matching or slightly beating DiLoCo in experiments.

  10. IRT-Router: Effective and Interpretable Multi-LLM Routing via Item Response Theory

    cs.AI 2025-06 conditional novelty 6.0 of 10

    An IRT-based router that models each LLM's latent ability and each query's difficulty outperforms RouterBench on cost-performance reward across ID and OOD benchmarks.

  11. Solving MNIST with a globally trained Mixture of Quantum Experts

    quant-ph 2025-05 conditional novelty 6.0 of 10

    A globally trained mixture of 16 quantum experts classifies full-resolution MNIST parity with 97.5% test accuracy using 10 qubits, and joint training improves compute-efficiency until saturation.

  12. Boosting Generalizable Depth Estimation in Endoscopy by Mixture of Lightweight Experts and Intrinsic Image Alignment

    cs.CV 2026-08 conditional novelty 5.0 of 10

    EndoMINI improves endoscopic depth estimation by combining mixture-of-experts LoRA fine-tuning with intrinsic image alignment losses, achieving state-of-the-art results on SCARED, Hamlyn, and SERV-CT.

  13. Double-Scoring: Reliable Extraction of Strong Lottery Tickets

    cs.LG 2026-07 conditional novelty 5.0 of 10

    Double-scoring replaces layerwise sparsity selection with fixed half-density masking over an augmented score tensor, yielding higher untrained-subnetwork accuracy but at uncontrolled effective sparsity.

  14. PROBE: Benchmarking Code Generation in Large Language Models

    cs.SE 2026-07 conditional novelty 5.0 of 10

    A multi-language evaluation framework measuring correctness, solution proximity, and code quality finds current LLMs pass at most ~0.70 per language and worsen sharply with problem difficulty.

  15. MoGU V2: Toward a Higher Pareto Frontier Between Model Usability and Security

    cs.CL 2025-09 conditional novelty 5.0 of 10

    MoGUv2 embeds small routers in the deeper layers of LLMs to dynamically blend a helpful variant and a refusal variant, improving safety against jailbreak and fine-tuning attacks while preserving usability.

  16. LLMs for LLMs: A Structured Prompting Methodology for Long Legal Documents

    cs.AI 2025-09 reject novelty 5.0 of 10

    On CUAD legal contracts, a prompt-engineered QWEN-2 pipeline with chunking and two answer-selection heuristics reportedly outperforms the fine-tuned DeBERTa-large baseline by about 9%, reaching claimed state-of-the-ar...

  17. LegalChainReasoner: A Legal Chain-guided Framework for Criminal Judicial Opinion Generation

    cs.CL 2025-08 conditional novelty 5.0 of 10

    A legal-chain-guided framework generates criminal judicial opinions, jointly producing legal reasoning and sentencing predictions, and outperforms baselines on two Chinese case datasets.

  18. DIME-Net: A Dual-Illumination Adaptive Enhancement Network Based on Retinex and Mixture-of-Experts

    cs.CV 2025-08 conditional novelty 5.0 of 10

    A single Retinex-based network with sparse mixture-of-experts tone curves, trained on a mixed low-light/backlit dataset, improves PSNR/SSIM/LPIPS on LOLv1 and BAID without dataset-specific retraining.

  19. CoCre-Sam (Kokkuri-san): Modeling Ouija Board as Collective Langevin Dynamics Sampling from Fused Language Models

    cs.MA 2025-07 conditional novelty 5.0 of 10

    Ouija board movement is modeled as collective Langevin dynamics sampling from a product-of-experts fusion of the participants' language models.

  20. Attentions Under the Microscope: A Comparative Study of Resource Utilization for Variants of Self-Attention

    cs.LG 2025-07 conditional novelty 5.0 of 10

    In a controlled GPT-2 fine-tuning benchmark, Flash Attention consumed the least GPU energy (1.07 MJ), followed closely by LSH Attention and Multi-Head Latent Attention.

  21. Identity-Preserving Text-to-Image Generation via Dual-Level Feature Decoupling and Expert-Guided Fusion

    cs.CV 2025-05 conditional novelty 5.0 of 10

    A dual-level decoupling module with contrastive losses and a mixture-of-experts fusion layer improves identity preservation in subject-driven text-to-image generation.

  22. Pangu Pro MoE: Mixture of Grouped Experts for Efficient Sparsity

    cs.CL 2025-05 conditional novelty 5.0 of 10

    A group-constrained Mixture-of-Experts routing rule (MoGE) is proposed and demonstrated in Pangu Pro MoE, a 72B/16B-active sparse LLM that reports faster inference on Ascend NPUs.

  23. Uni3D-MoE: Scalable Multimodal 3D Scene Understanding via Mixture of Experts

    cs.CV 2025-05 conditional novelty 5.0 of 10

    A sparse mixture-of-experts 3D multimodal LLM adaptively fuses RGB, RGBD, BEV, point cloud, and voxel tokens, achieving SOTA on several ScanNet-based 3D scene understanding benchmarks.

  24. Sci-LoRA: Mixture of Scientific LoRAs for Cross-Domain Lay Paraphrasing

    cs.CL 2025-05 conditional novelty 5.0 of 10

    Sci-LoRA dynamically mixes domain-specific LoRA adapters and achieves state-of-the-art lay paraphrasing across twelve domains without needing domain labels at inference.

  25. DualComp: End-to-End Learning of a Unified Dual-Modality Lossless Compressor

    cs.CV 2025-05 conditional novelty 5.0 of 10

    A unified RWKV-based model compresses both images and text losslessly with a shared vocabulary, modality-specific contextual layers, and mixture-of-experts routing, matching or improving on much larger SOTA compressor...

  26. An Evaluation of Large Language Models on Text Summarization Tasks Using Prompt Engineering Techniques

    cs.CL 2025-07 conditional novelty 4.0 of 10

    A broad benchmark of six open-weights LLMs shows prompt design and chunking affect summarization quality more than model size alone.

  27. Towards Accurate and Efficient 3D Object Detection for Autonomous Driving: A Mixture of Experts Computing System on Edge

    cs.CV 2025-07 reject novelty 4.0 of 10

    EMC2 routes each driving scene to one of three expert 3D detectors using distance and confidence thresholds, reporting higher accuracy and 2.6x faster inference on Jetson, though thresholds are tuned on the validation set.

  28. GigaChat Family: Efficient Russian Language Modeling Through Mixture of Experts Architecture

    cs.CL 2025-06 conditional novelty 4.0 of 10

    A from-scratch Russian MoE LLM family is released with competitive benchmark scores, but key efficiency and novelty claims are not fully evidenced.

  29. Towards Efficient Multi-LLM Inference: Characterization and Analysis of LLM Routing and Hierarchical Techniques

    cs.LG 2025-06 unverdicted novelty 4.0 of 10

    A survey of LLM routing and hierarchical inference techniques that proposes an unvalidated unified evaluation metric called the Inference Efficiency Score.

  30. SkipGPT: Dynamic Layer Pruning Reinvented with Token Awareness and Module Decoupling

    cs.CL 2025-06 conditional novelty 4.0 of 10

    SkipGPT learns per-token, per-module routers that dynamically skip attention and MLP blocks, then uses LoRA to recover most of the original model's quality.

  31. Position: Agentic Systems Constitute a Key Component of Next-Generation Intelligent Image Processing

    cs.CV 2025-05 conditional novelty 4.0 of 10

    Image processing should move from monolithic deep models to agentic systems that orchestrate multiple tools, with a proposed six-level autonomy ladder.

  32. GPT-OSS-20B: A Comprehensive Deployment-Centric Analysis of OpenAI's Open-Weight Mixture of Experts Model

    cs.AR 2025-08 conditional novelty 3.0 of 10

    On a single H100, the 20.9B-parameter MoE model GPT-OSS-20B shows roughly 32% higher decode throughput, 26% lower energy per 1,000 tokens, and 32% lower peak VRAM than dense Qwen3-32B at 2,048-token context, at the co...

Pith tools