Pith. sign in

REVIEW 15 cited by

Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2309.05444 v1 pith:ND2WWIX3 submitted 2023-09-11 cs.CL cs.LG

Ted Zadouri , Ahmet Üstün , Arash Ahmadian , Beyza Ermiş , Acyr Locatelli , Sara Hooker This is my paper · ORCID
classification cs.CLcs.LG
keywords expertsarchitecturemixtureextremelyfine-tuninglightweightlimitparameter
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

The Mixture of Experts (MoE) is a widely known neural architecture where an ensemble of specialized sub-models optimizes overall performance with a constant computational cost. However, conventional MoEs pose challenges at scale due to the need to store all experts in memory. In this paper, we push MoE to the limit. We propose extremely parameter-efficient MoE by uniquely combining MoE architecture with lightweight experts.Our MoE architecture outperforms standard parameter-efficient fine-tuning (PEFT) methods and is on par with full fine-tuning by only updating the lightweight experts -- less than 1% of an 11B parameters model. Furthermore, our method generalizes to unseen tasks as it does not depend on any prior task knowledge. Our research underscores the versatility of the mixture of experts architecture, showcasing its ability to deliver robust performance even when subjected to rigorous parameter constraints. Our code used in all the experiments is publicly available here: https://github.com/for-ai/parameter-efficient-moe.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 15 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. FlexOlmo: Open Language Models for Flexible Data Use

    cs.CL 2025-07 conditional novelty 7.0 of 10

    FlexOlmo merges independently trained language-model experts, trained on private data, into a single mixture-of-experts model without joint training.

  2. SAM+D: Parameter-Efficient Dimensional Lifting of SAM-Family Models via Depth-Routed LoRA and Depth Shifting

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Depth-routed LoRA and a depth-shift module lift frozen SAM and SAM2 to 3D and 3D+T segmentation using less than ~3.7% trainable parameters.

  3. CoMoL: Efficient Mixture of LoRA Experts via Dynamic Core Space Merging

    cs.CL 2026-02 conditional novelty 6.0 of 10

    CoMoL represents every LoRA expert as a shared-basis core matrix and merges token-selected experts in that core space, reaching standard LoRA parameter counts while outperforming MoE-LoRA baselines on math and code.

  4. Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient

    cs.LG 2025-02 conditional novelty 6.0 of 10

    A joint scaling law fitted to over 280 models shows that, under fixed memory or total-parameter budgets, MoE models can achieve lower loss than dense models when trained on more tokens.

  5. Each Rank Could be an Expert: Single-Ranked Mixture of Experts LoRA for Multi-Task Learning

    cs.LG 2025-01 conditional novelty 6.0 of 10

    SMoRA embeds a sparse mixture of experts inside a single LoRA by routing over individual ranks, and reports consistent (though small) multi-task gains over LoRA and LoRA-MoE baselines.

  6. CLONE: Customizing LLMs for Efficient Latency-Aware Inference at the Edge

    cs.AR 2025-06 conditional novelty 5.0 of 10

    CLONE combines generative pruning, LoRA adapters, a parameter-free Mixture-of-Experts router, and learning-based DVFS to make LLM inference on edge devices faster and more energy-efficient, claiming up to 11.92x speed...

  7. A Sensitivity-Driven Expert Allocation Method in LoRA-MoE for Efficient Fine-Tuning

    cs.LG 2025-05 conditional novelty 5.0 of 10

    Gradient-based sensitivity scoring automatically allocates LoRA-MoE expert budget across parameter blocks; the module-separated variant matches or slightly beats prior methods with fewer trainable parameters.

  8. PointLoRA: Low-Rank Adaptation with Token Selection for Point Cloud Learning

    cs.CV 2025-04 conditional novelty 5.0 of 10

    PointLoRA combines low-rank adaptation with multi-scale token selection to fine-tune point cloud transformers using only 3.43% trainable parameters while matching or exceeding full fine-tuning on benchmark tasks.

  9. Superposition in Transformers: A Novel Way of Building Mixture of Experts

    cs.CL 2024-12 conditional novelty 5.0 of 10

    Merging two GPT-2 variants with B-spline-blended hidden states plus autoencoders yields a single model that keeps both English and French perplexity closer to the best expert than linear interpolation.

  10. ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing

    cs.LG 2024-12 conditional novelty 5.0 of 10

    ReLU-based routing with adaptive L1 regularization outperforms TopK routing in MoE language models across model sizes, expert counts, and granularity.

  11. Large Language Models for Scholarly Ontology Generation: An Extensive Analysis in the Engineering Field

    cs.DL 2024-12 conditional novelty 5.0 of 10

    Zero-shot LLMs, especially Claude 3 Sonnet and a fine-tuned 7B Mistral variant, classify semantic relations between engineering research topics with high F1 on the new IEEE-Rel-1K benchmark.

  12. PERFT: Parameter-Efficient Routed Fine-Tuning for Mixture-of-Expert Model

    cs.LG 2024-11 conditional novelty 5.0 of 10

    PERFT inserts routed, parameter-efficient LoRA experts into MoE layers and reports consistent gains over attention-only LoRA on reasoning benchmarks.

  13. Generative AI for Character Animation: A Comprehensive Survey of Techniques, Applications, and Future Directions

    cs.CV 2025-04 conditional novelty 3.0 of 10

    A comprehensive survey that unifies generative AI techniques for character animation across facial, gesture, motion, and 3D asset generation, with a shared taxonomy and resource list.

  14. Mixture of Experts (MoE): A Big Data Perspective

    cs.LG 2025-01 conditional novelty 2.0 of 10

    A survey of MoE methods for big data that catalogs architectures, use cases, and open challenges without adding new results.

  15. Survey of different Large Language Model Architectures: Trends, Benchmarks, and Challenges

    cs.LG 2024-12 conditional

    A broad but error-prone survey of LLM and MLLM architectures, training methods, benchmarks, and challenges.

Pith tools