Pith. sign in

REVIEW 21 cited by

GLaM: Efficient Scaling of Language Models with Mixture-of-Experts

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2112.06905 v2 pith:X55NRAQW submitted 2021-12-13 cs.CL

Nan Du , Yanping Huang , Andrew M. Dai , Simon Tong , Dmitry Lepikhin , Yuanzhong Xu , Maxim Krikun , Yanqi Zhou
show 19 more authors
Adams Wei Yu Orhan Firat Barret Zoph Liam Fedus Maarten Bosma Zongwei Zhou Tao Wang Yu Emma Wang Kellie Webster Marie Pellat Kevin Robinson Kathleen Meier-Hellstern Toju Duke Lucas Dixon Kun Zhang Quoc V Le Yonghui Wu Zhifeng Chen Claire Cui
This is my paper
classification cs.CL
keywords languagemodelsglamgpt-3scalingdensemixture-of-expertsmodel
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Scaling language models with more data, compute and parameters has driven significant progress in natural language processing. For example, thanks to scaling, GPT-3 was able to achieve strong results on in-context learning tasks. However, training these large dense models requires significant amounts of computing resources. In this paper, we propose and develop a family of language models named GLaM (Generalist Language Model), which uses a sparsely activated mixture-of-experts architecture to scale the model capacity while also incurring substantially less training cost compared to dense variants. The largest GLaM has 1.2 trillion parameters, which is approximately 7x larger than GPT-3. It consumes only 1/3 of the energy used to train GPT-3 and requires half of the computation flops for inference, while still achieving better overall zero-shot and one-shot performance across 29 NLP tasks.

Discussion (0). Sign in to comment.

Forward citations

Cited by 21 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Expert Upcycling: Shifting the Compute-Efficient Frontier of Mixture-of-Experts

    cs.LG 2026-04 unverdicted novelty 7.0 of 10

    Expert upcycling duplicates experts in an existing MoE checkpoint and continues pre-training to match fixed-size baseline performance with 32% less compute.

  2. The Ray Tracing Sampler: Bayesian Sampling of Neural Networks for Everyone

    astro-ph.IM 2025-10 conditional novelty 7.0 of 10

    A new ray-tracing MCMC sampler keeps ray speed constant, making it far more robust to stochastic gradients and able to sample billion-parameter neural networks on one GPU.

  3. ASAP: A Disaggregated and Asynchronous Inference System for MoE Prefill

    cs.DC 2026-06 unverdicted novelty 6.0 of 10

    ASAP is a disaggregated asynchronous inference system for the prefill phase of MoE models that removes DP-EP synchronization barriers and reports 90% higher SLO-compliant throughput than synchronous baselines.

  4. Hyperbolic and Evidence-Prioritized Experts for Large Vision-Language Models

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    AsyMoE adds hyperbolic geometry for cross-modal hierarchies and evidence-priority experts to address vision-language asymmetry in LVLMs, reporting 1.5% average gains and 25.45% fewer active parameters.

  5. ROMER: Expert Replacement and Router Calibration for Robust MoE LLMs on Analog Compute-in-Memory Systems

    cs.LG 2026-05 conditional novelty 6.0 of 10

    ROMER cuts perplexity by up to 59% in noisy analog CIM environments for MoE LLMs via expert replacement and router recalibration calibrated on real-chip measurements.

  6. A Meta Reinforcement Learning Approach to Goals-Based Wealth Management

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    MetaRL pre-trained on GBWM problems delivers near-optimal dynamic strategies in 0.01s achieving 97.8% of DP optimal utility and handles larger problems where DP fails.

  7. Expert Upcycling: Shifting the Compute-Efficient Frontier of Mixture-of-Experts

    cs.LG 2026-04 unverdicted novelty 6.0 of 10

    Expert upcycling expands MoE models by duplicating experts and continuing pre-training, matching baseline performance while saving 32% GPU hours in 7B-13B experiments.

  8. Breaking the MoE LLM Trilemma: Dynamic Expert Clustering with Structured Compression

    cs.CL 2025-09 conditional novelty 6.0 of 10

    Dynamic expert clustering plus shared-base, low-rank residual compression and two-stage routing cuts MoE parameter count by about 80% with a modest quality cost on GLUE and WikiText-103.

  9. DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models

    cs.LG 2023-09 accept novelty 6.0 of 10

    DeepSpeed-Ulysses keeps communication volume constant for sequence-parallel attention when sequence length and device count scale together, delivering 2.5x faster training on 4x longer sequences than prior SOTA.

  10. Efficient Training of Language Models to Fill in the Middle

    cs.CL 2022-07 unverdicted novelty 6.0 of 10

    Autoregressive language models trained on data with middle spans relocated to the end learn infilling without degrading left-to-right perplexity or sampling quality.

  11. Scaling Autoregressive Models for Content-Rich Text-to-Image Generation

    cs.CV 2022-06 unverdicted novelty 6.0 of 10

    Scaling an autoregressive Transformer to 20B parameters for text-to-image generation using image token sequences achieves new SOTA zero-shot FID of 7.23 and fine-tuned FID of 3.22 on MS-COCO.

  12. Emergent Abilities of Large Language Models

    cs.CL 2022-06 unverdicted novelty 6.0 of 10

    Emergent abilities are capabilities present in large language models but absent in smaller ones and cannot be predicted by extrapolating smaller model performance.

  13. PaLM: Scaling Language Modeling with Pathways

    cs.CL 2022-04 accept novelty 6.0 of 10

    PaLM 540B demonstrates continued scaling benefits by setting new few-shot SOTA results on hundreds of benchmarks and outperforming humans on BIG-bench.

  14. Safe and Certifiable AI Systems: Concepts, Challenges, and Lessons Learned

    cs.CY 2025-09 conditional novelty 5.0 of 10

    The paper presents the TÜV AUSTRIA Trusted AI audit catalog, a statistical framework based on the Stochastic Application Domain Definition, minimum performance requirements, and independent-sample testing for certifyi...

  15. Apple Intelligence Foundation Language Models: Tech Report 2025

    cs.LG 2025-07 conditional novelty 5.0 of 10

    Apple's 3B on-device and larger server language models match or beat several similarly sized open models on MMLU, MMMLU, and MGSM, using new efficiency techniques like KV-cache sharing and 2-bit quantization.

  16. AbbIE: Autoregressive Block-Based Iterative Encoder for Efficient Sequence Modeling

    cs.LG 2025-07 conditional novelty 5.0 of 10

    An iterative Transformer that reuses its middle layers with an extra residual connection matches a standard Transformer after one pass and improves perplexity and zero-shot accuracy when run for more passes at test time.

  17. PaLM 2 Technical Report

    cs.CL 2023-05 unverdicted novelty 5.0 of 10

    PaLM 2 reports state-of-the-art results on language, reasoning, and multilingual tasks with improved efficiency over PaLM.

  18. Memory for Large Language Models

    cs.CL 2026-07 conditional novelty 4.0 of 10

    The paper organizes LLM memory architectures along three axes—implicit vs explicit representation, offline vs online update, and short vs long-term persistence—plus finer-grained update rules.

  19. Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities

    cs.CL 2025-07 unverdicted novelty 4.0 of 10

    Gemini 2.5 Pro and Flash models are presented as achieving frontier performance in reasoning, coding, and long-context multimodal tasks while spanning a cost-capability Pareto curve.

  20. A Vision Toward Energy-Efficient Domain-Specific Artificial Intelligence Models and Agents

    cs.AI 2025-10 unverdicted novelty 2.0 of 10

    A position paper proposing compact, domain-specific AI agents as the path to ≥1000× energy efficiency, without demonstrating the claim.

  21. The Carbon Cost of Conversation, Sustainability in the Age of Language Models

    cs.CY 2025-07 reject novelty 2.0 of 10

    A review-style essay that claims LLM training and use are environmentally expensive, but the quantitative evidence it presents is unreliable and internally inconsistent.

Pith tools