REVIEW 17 cited by
GLaM: Efficient Scaling of Language Models with Mixture-of-Experts
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Scaling language models with more data, compute and parameters has driven significant progress in natural language processing. For example, thanks to scaling, GPT-3 was able to achieve strong results on in-context learning tasks. However, training these large dense models requires significant amounts of computing resources. In this paper, we propose and develop a family of language models named GLaM (Generalist Language Model), which uses a sparsely activated mixture-of-experts architecture to scale the model capacity while also incurring substantially less training cost compared to dense variants. The largest GLaM has 1.2 trillion parameters, which is approximately 7x larger than GPT-3. It consumes only 1/3 of the energy used to train GPT-3 and requires half of the computation flops for inference, while still achieving better overall zero-shot and one-shot performance across 29 NLP tasks.
Forward citations
Cited by 17 Pith papers
-
The Ray Tracing Sampler: Bayesian Sampling of Neural Networks for Everyone
A new ray-tracing MCMC sampler keeps ray speed constant, making it far more robust to stochastic gradients and able to sample billion-parameter neural networks on one GPU.
-
Breaking the MoE LLM Trilemma: Dynamic Expert Clustering with Structured Compression
Dynamic expert clustering plus shared-base, low-rank residual compression and two-stage routing cuts MoE parameter count by about 80% with a modest quality cost on GLUE and WikiText-103.
-
Kinetics: Rethinking Test-Time Scaling Laws
A memory-aware test-time scaling law shows small models are overestimated and sparse attention is needed for efficient scaling.
-
M2R2: Mixture of Multi-Rate Residuals for Efficient Transformer Inference
M2R2 trains a parallel accelerated residual stream to align with the base model at a faster rate, improving early-exit, self-speculative decoding, and MoE expert prefetching.
-
Safe and Certifiable AI Systems: Concepts, Challenges, and Lessons Learned
The paper presents the TÜV AUSTRIA Trusted AI audit catalog, a statistical framework based on the Stochastic Application Domain Definition, minimum performance requirements, and independent-sample testing for certifyi...
-
Apple Intelligence Foundation Language Models: Tech Report 2025
Apple's 3B on-device and larger server language models match or beat several similarly sized open models on MMLU, MMMLU, and MGSM, using new efficiency techniques like KV-cache sharing and 2-bit quantization.
-
AbbIE: Autoregressive Block-Based Iterative Encoder for Efficient Sequence Modeling
An iterative Transformer that reuses its middle layers with an extra residual connection matches a standard Transformer after one pass and improves perplexity and zero-shot accuracy when run for more passes at test time.
-
Scaling Fine-Grained MoE Beyond 50B Parameters: Empirical Evaluation and Practical Insights
At 56B total parameters, fine-grained MoE with smaller, more numerous experts beats standard Switch and Mixtral-style MoE on validation loss and average downstream accuracy at matched FLOPs.
-
Rethinking Data Mixture for Large Language Models: A Comprehensive Survey and New Perspectives
A survey organizing LLM data mixture methods into offline and online families, with a fine-grained taxonomy based on optimization frameworks.
-
CLIP-UP: A Simple and Efficient Mixture-of-Experts CLIP Training Recipe with Sparse Upcycling
CLIP-UP converts a pre-trained dense CLIP into an MoE model and improves zero-shot text-image retrieval beyond dense baselines at lower inference cost.
-
Memory for Large Language Models
The paper organizes LLM memory architectures along three axes—implicit vs explicit representation, offline vs online update, and short vs long-term persistence—plus finer-grained update rules.
-
Personalized Artificial General Intelligence (AGI) via Neuroscience-Inspired Continuous Learning Systems
The paper proposes a theoretical Tri-Memory architecture that combines Hebbian updates, pruning, replay, and sparse coding for lifelong personalized learning on edge devices, without empirical validation.
-
Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models
For MoE language models, under a fixed training compute budget, increasing sparsity while adding total parameters lowers pretraining loss, and the optimal sparsity grows with model size.
-
Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Models
This survey organizes LLM synthetic data research around quality, diversity, and complexity, claiming quality mainly helps in-distribution generalization, diversity mainly helps out-of-distribution generalization, and...
-
MH-MoE: Multi-Head Mixture-of-Experts
A recipe for setting MH-MoE expert widths, counts, and top-k to match SMoE FLOPs is presented with small perplexity gains, but the recipe's core equation is wrong and the experimental parity is not exact.
-
A Vision Toward Energy-Efficient Domain-Specific Artificial Intelligence Models and Agents
A position paper proposing compact, domain-specific AI agents as the path to ≥1000× energy efficiency, without demonstrating the claim.
-
The Carbon Cost of Conversation, Sustainability in the Age of Language Models
A review-style essay that claims LLM training and use are environmentally expensive, but the quantitative evidence it presents is unreliable and internally inconsistent.
Discussion (0). Continue with ORCID to comment.