Pith. sign in

REVIEW 10 cited by

LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.16554 v1 pith:ZSDM76VO submitted 2024-06-24 cs.CL

Tong Zhu , Xiaoye Qu , Daize Dong , Jiacheng Ruan , Jingqi Tong , Conghui He , Yu Cheng This is my paper
classification cs.CL
keywords modelscontinuallanguagellama-moemodelparameterspre-trainingbuilding
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Mixture-of-Experts (MoE) has gained increasing popularity as a promising framework for scaling up large language models (LLMs). However, training MoE from scratch in a large-scale setting still suffers from data-hungry and instability problems. Motivated by this limit, we investigate building MoE models from existing dense large language models. Specifically, based on the well-known LLaMA-2 7B model, we obtain an MoE model by: (1) Expert Construction, which partitions the parameters of original Feed-Forward Networks (FFNs) into multiple experts; (2) Continual Pre-training, which further trains the transformed MoE model and additional gate networks. In this paper, we comprehensively explore different methods for expert construction and various data sampling strategies for continual pre-training. After these stages, our LLaMA-MoE models could maintain language abilities and route the input tokens to specific experts with part of the parameters activated. Empirically, by training 200B tokens, LLaMA-MoE-3.5B models significantly outperform dense models that contain similar activation parameters. The source codes and models are available at https://github.com/pjlab-sys4nlp/llama-moe .

Discussion (0). Sign in to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Relax Within, Balance Across: Geometry-Guided Load Balancing for Vision-Language Mixture-of-Experts

    cs.CV 2026-08 accept novelty 6.0 of 10

    The standard mixed image-text load-balancing loss lets modality errors cancel at one token mix; ReBA balances each modality separately and per image, reducing load across resolutions and tiles with comparable accuracy.

  2. Communication-Aware Placement and Pruning for Efficient Mixture-of-Experts Inference

    cs.DC 2026-07 conditional novelty 6.0 of 10

    Communication-aware expert placement plus device-level pruning yields 1.23–1.86× MoE inference throughput and better accuracy at equal speedup than load-balance or sequential baselines.

  3. STAR: Rethinking MoE Routing as Structure-Aware Subspace Learning

    cs.AI 2026-06 unverdicted novelty 6.0 of 10

    STAR rethinks MoE routing as structure-aware subspace learning by adding a GHA-tracked principal subspace to standard routers, yielding more stable specialization and better performance on synthetic, language, and vis...

  4. Hierarchical Mixture-of-Experts with Two-Stage Optimization

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    Hi-MoE uses two-level hierarchical routing objectives to enforce group-level balance while promoting within-group specialization, yielding better perplexity and expert utilization than prior MoE baselines in NLP and v...

  5. SpaceMoE: Realizing Distributed Mixture-of-Experts Inference over Space Networks

    cs.DC 2026-05 unverdicted novelty 6.0 of 10

    SpaceMoE partitions MoE layers across orbiting satellite subnets in a ring and optimizes expert placement by activation probability and path latency, yielding at least 3x lower inference latency in thousand-satellite ...

  6. SpaceMoE: Realizing Distributed Mixture-of-Experts Inference over Space Networks

    cs.DC 2026-05 unverdicted novelty 6.0 of 10

    Space-XNet partitions satellite constellations into ring subnets for MoE layers and maps high-activation experts to low-latency satellites, yielding at least 3x lower inference latency than random or ablation placemen...

  7. Cache Management for Mixture-of-Experts LLMs -- extended version

    cs.LG 2025-09 conditional novelty 6.0 of 10

    The paper formalizes expert caching in MoE LLMs as layered paging, proves near-tight competitive lower bounds, and shows a layer-aware LRU variant beats standard LRU on real traces.

  8. What You Have is What You Track: Adaptive and Robust Multimodal Tracking

    cs.CV 2025-07 conditional novelty 6.0 of 10

    FlexTrack claims state-of-the-art multimodal tracking on complete and simulated missing-modality benchmarks, using heterogeneous mixture-of-experts fusion and a video-level masking training strategy.

  9. Capacity-Aware Inference: Mitigating the Straggler Effect in Mixture of Experts

    cs.LG 2025-03 conditional novelty 6.0 of 10

    Capacity-aware dropping techniques mitigate load imbalance in MoE inference, delivering up to 1.85x speedup with 0.2% or less performance change on models including Mixtral-8x7B.

  10. Joint Learning using Mixture-of-Expert-Based Representation for Speech Enhancement and Robust Emotion Recognition

    eess.AS 2025-09 unverdicted novelty 5.0 of 10

    Sparse MERIT uses frame-wise sparse mixture-of-experts with task-specific gating on self-supervised speech features to jointly optimize enhancement and emotion recognition, reporting gains over baselines on MSP-Podcas...

Pith tools