Pith. sign in

REVIEW 22 cited by

Mixture of A Million Experts

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2407.04153 v1 pith:RPK6NES4 submitted 2024-07-04 cs.LG cs.AI

Xu Owen He This is my paper · ORCID
classification cs.LGcs.AI
keywords computationalexpertspeerarchitecturesefficientlayerlayersmillion
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

The feedforward (FFW) layers in standard transformer architectures incur a linear increase in computational costs and activation memory as the hidden layer width grows. Sparse mixture-of-experts (MoE) architectures have emerged as a viable approach to address this issue by decoupling model size from computational cost. The recent discovery of the fine-grained MoE scaling law shows that higher granularity leads to better performance. However, existing MoE models are limited to a small number of experts due to computational and optimization challenges. This paper introduces PEER (parameter efficient expert retrieval), a novel layer design that utilizes the product key technique for sparse retrieval from a vast pool of tiny experts (over a million). Experiments on language modeling tasks demonstrate that PEER layers outperform dense FFWs and coarse-grained MoEs in terms of performance-compute trade-off. By enabling efficient utilization of a massive number of experts, PEER unlocks the potential for further scaling of transformer models while maintaining computational efficiency.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 22 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Monet: Mixture of Monosemantic Experts for Transformers

    cs.AI 2024-12 conditional novelty 8.0 of 10

    Monet composes experts from shared sub-layers to scale to 262,144 experts per layer with parameter growth proportional to the square root of the expert count, yielding more monosemantic and manipulable LLMs.

  2. The power of fine-grained experts: Granularity boosts expressivity in Mixture of Experts

    cs.LG 2025-05 conditional novelty 7.0 of 10

    For MoE layers with equal active parameters, increasing the number of active experts from k' to k yields an exponential separation in L2 approximation error whenever the binomial coefficient binom(m,k) dominates binom(m',k').

  3. Training, Reading, and Editing Legible Transformers

    cs.LG 2026-07 conditional novelty 6.5 of 10

    A variance-floor objective plus learned operator gates produce an end-to-end legible transformer whose crisp units are 50–184× more local to edit and can be reshaped from fan-out to fan-in circuits without quality loss.

  4. OmniMoE: An Efficient MoE by Orchestrating Atomic Experts at Scale

    cs.CL 2026-02 conditional novelty 6.0 of 10

    OmniMoE's vector-level atomic experts + Cartesian-product router + expert-centric scheduling gives 50.9% zero-shot accuracy and a 10.9x inference speedup over PEER at matched 1.7B active parameters.

  5. Towards Foundational Models for Dynamical System Reconstruction: Hierarchical Meta-Learning via Mixture of Experts

    cs.LG 2025-02 conditional novelty 6.0 of 10

    MixER uses K-means clustering and least squares to route each dynamical system to a specialist expert, improving reconstruction across loosely related ODE families in low-data regimes while underperforming on closely ...

  6. Accelerating Linear Recurrent Neural Networks for the Edge with Unstructured Sparsity

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Sparse, 8-bit quantized S5 linear RNNs match dense model audio denoising accuracy with 2x less compute and 36% less memory, and run 42x faster with 149x lower energy on Loihi 2 than a dense FP32 model on Jetson Orin Nano.

  7. Memory Layers at Scale

    cs.CL 2024-12 conditional novelty 6.0 of 10

    A scaled-up, shared, gated memory layer improves factual recall in language models and can match or beat denser models trained with much more compute.

  8. Mixture of Hidden-Dimensions Transformer

    cs.CL 2024-12 conditional novelty 6.0 of 10

    MoHD selectively activates shared and token-specific sub-dimensions of the hidden state, matching or beating dense Transformers with fewer active parameters.

  9. Ultra-Sparse Memory Network

    cs.LG 2024-11 conditional novelty 6.0 of 10

    UltraMem, a sparse memory layer with Tucker-decomposed retrieval and virtual memory expansion, outperforms Mixture-of-Experts at equal compute, with up to 6x lower inference latency.

  10. The SpiNNaker2 chip: a many-core platform for flexible and scalable brain-inspired computing

    cs.ET 2026-07 accept novelty 5.5 of 10

    SpiNNaker2 delivers a measured many-core platform combining ARM cores, ML accelerators, and event routing that runs SNNs, DNNs, and hybrid event-based models on one scalable chip.

  11. More Than Memory: Task-Conditioned Signed FFN Writes in Long-Context Retrieval

    cs.LG 2026-06 conditional novelty 5.0 of 10

    Scaling a model's own per-layer feed-forward writes shows signed, task-dependent retrieval roles, and the sign of the write–gradient dot product identifies suppressors from amplifiers.

  12. MoLEx: Mixture of LoRA Experts in Speech Self-Supervised Models for Audio Deepfake Detection

    cs.SD 2025-09 conditional novelty 5.0 of 10

    MoLEx combines LoRA adapters with a top-K expert router inside a frozen WavLM model, achieving 5.56% EER on ASVSpoof 5 without augmentation.

  13. PC-MoE: Memory-Efficient and Privacy-Preserving Collaborative Training for Mixture-of-Experts LLMs

    cs.LG 2025-06 conditional novelty 5.0 of 10

    PC-MoE shards the expert layers of an MoE LLM across parties and routes only sparse top-k activations between them, achieving near-centralized accuracy with about 70% memory savings and resistance to one partial-gradi...

  14. EfficientLLM: Efficiency in Large Language Models

    cs.CL 2025-05 conditional novelty 5.0 of 10

    A large-scale benchmark of LLM efficiency techniques finds that every method trades off one resource for another, with the best choice depending on model scale, task, and hardware.

  15. ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing

    cs.LG 2024-12 conditional novelty 5.0 of 10

    ReLU-based routing with adaptive L1 regularization outperforms TopK routing in MoE language models across model sizes, expert counts, and granularity.

  16. Pro-Prophet: A Systematic Load Balancing Method for Efficient Parallel Training of Large-scale MoE Models

    cs.DC 2024-11 conditional novelty 5.0 of 10

    Pro-Prophet balances MoE training load by moving each expert to only the devices that need it and by pre-scheduling those moves to hide communication behind computation.

  17. The Evolution of Mixture-of-Experts Architectures in Large Language Models: Routing, Topology, Load Balancing, and Expert Parallelism

    cs.CL 2026-08 conditional novelty 4.0 of 10

    A survey proposes eight architectural milestones and four control planes for MoE LLMs, arguing the field is moving toward decoupling routing, compute budgets, and physical execution.

  18. UltraMemV2: Memory Networks Scaling to 120B Parameters with Superior Long-Context Learning

    cs.LG 2025-08 conditional novelty 4.0 of 10

    A redesigned memory-layer architecture with five engineering improvements reaches performance parity with 8-expert MoE at similar compute, with lower memory access and stronger long-context memorization.

  19. Scaling and Enhancing LLM-based AVSR: A Sparse Mixture of Projectors Approach

    eess.AS 2025-05 conditional novelty 4.0 of 10

    Llama-SMoP-DEDR, a sparse mixture of projectors with modality-specific experts and routers, lowers word error rate for LLM-based AVSR on LRS3, mainly with smaller LLMs.

  20. Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models

    cs.LG 2025-01 conditional novelty 4.0 of 10

    For MoE language models, under a fixed training compute budget, increasing sparsity while adding total parameters lowers pretraining loss, and the optimal sparsity grows with model size.

  21. Wonderful Matrices: Combining for a More Efficient and Effective Foundation Model Architecture

    cs.LG 2024-12 conditional novelty 4.0 of 10

    A hybrid SSD-attention-MoE architecture, Cheems, is reported to outperform LLaMA3, Mamba2, and Jamba on matched-size language-model evaluations, with a proposed RoPE-for-SSD proof and a new dynamic-mask attention.

  22. Mixture of Experts (MoE): A Big Data Perspective

    cs.LG 2025-01 conditional novelty 2.0 of 10

    A survey of MoE methods for big data that catalogs architectures, use cases, and open challenges without adding new results.

Pith tools